ERK'2019, Portorož, 110-113 110 Combining Foot Placement Prediction with Obstacle Detection to Detect Tripping Urban Bobek 1 , Elmar Rueckert 3 , Marko Jamˇ sek 1;2 , Saˇ sa Bariˇ si´ c 4 , Jan Babiˇ c 1 1 Laboratory for Neuromechanics and Biorobotics, Department of Automation, Biocybernetics and Robotics 2 Jozef Stefan International Postgraduate School, Jamova cesta 39, 1000 Ljubljana, Slovenia 3 Institute for Robotics and Cognitive Systems, University of Luebeck, Ratzeburger Allee 160, 23562 Luebeck, Germany 4 Bjelovar University of Applied Sciences,Trg Eugena Kvaternika 4, 43000 Bjelovar, Croatia E-poˇ sta: bobek.urban@gmail.com Abstract Tripping is a major cause of fall related injuries, espe- cialy among the elderly population. Some reaserch has been done on the mechanics of tripping and strategies to gain balance afterwards. But what if you could detect a potential trip in advance and possibly prevent it? We pro- pose a system that involves detecting obstacles infront of the user and a method to predict whether they will hit it. 1 Introduction Falls and fall related injuries are very common among people, especially in the elderly population [1]. This is becoming an ever increasing problem, because the amount of elderly people is getting larger with every year [2]. The mechanism of tripping have been explored in the past [3, 4], but here we are interested in preventing tripping from even occuring. For this we would need a way to predict the future placement of the subjects feet and a prediction system of obstacles. Some research has already been done on predicting foot placement. In [5] they predicted foot placement in the mediolater direction based on the position and veloc- ity of the subjects center of mass. Similarly, [6] predicted recovery foot placement when pertubating a subject in the sagital plane. A method, where the next contact lo- cation of the subjects foot gets predicted needed to be made. The prediction method we implemented takes ad- vantage of the properties of probabilistic motion primi- tives (ProMPs) [7], that allow operations from probability theory. ProMPs have been used in many different robotic applications [8, 9] and also postural studies [10] to en- code and predict movements. We combined this method with an obstacle detection system mounted on the sub- ject. In this article we present a system that can both detect potential obstacles infront of the wearer and also predict if there is a chance of tripping. 2 Equipment and setup The entire system consist of two major parts, the foot placement prediction and the obstacle detection. The lat- ter was achieved with the combination of a depth cam- era (RealSense Depth Camera D435, Intel, Santa Clara, USA) and 3D motion capture system (3D Investigator, NDI, Waterloo, Canada). For the foot placement pre- diction, an Xsens motion capture suite (MTw Awinda, Xsens, Enschede, Netherlands) was used to capture sub- jects walking gait. Using this data we could then online predict where the subject is going to step using methods described in Section 3. A block diagram of the whole setup is depicted in Figure 1). The main computer re- ceives the foot placement predictions from the Xsens com- puter, the data from the RealSense camera and the 3D motion capture system. It then combines this data to cal- culate if the subject is in danger of tripping. Xsens motion capture suit RealSense depth camera 3D Investigator Motion Capture XSens computer Main computer Motion capture data Foot placement prediction Video and point cloud Position of camera markers Figure 1: Block diagram of the setup used in the experiment. 2.1 Motion Capture In the setup we used two diffent motion capture (MC) systems. The first one is the Awinda human motion cap- ture system (MTw Awinda, Xsens, Enschede, Netherlands) that uses IMUs placed on certain parts of the body. Using a complex kinematic model it then returns the whole body kinematics of the wearer. Using the proprietary software we could stream the data at 60Hz to a local Simulink (Mathworks, Nat- ick, MA, United States) scheme. For this specific experiment, we were only interested in the position and velocity of the left and right ankle. The Simulink scheme then calculated the most probable foot placement of the subject and sent it foward to the main computer. The other MC system used was the 3D Investigator Motion Capture system (3D Investigator, NDI, Waterloo, Canada) that can track special markers in 3D space with 111 an accuracy of 0.4 mm. We used the RealSense depth camera to detect obstacles infront of the subject. Point cloud data captured by the camera is relative to its own coordinate frame, so we needed a way to know the cam- eras position and orientation in the global coordinate frame. The camera was attached to the subjects front and on it we placed 3 markers. Using the position of these markers we could then transform the depth camera data to the global frame. Both the depth camera and the motion tracking system streamed their data to the main computer, where all of the calculation necesarry was done. 2.2 Depth Camera for Obstacle Detection The RealSense depth camera was used for object detec- tion. We used both the RGB video and the point cloud output of the camera to help represent the results on the main computer. The camera was slanted at an angle (ap- prox. 40 ), so that the floor infront of the subject was visible. 3 Methods In this section we will describe the general theory of how the foot placement was predicted using probabilistic mo- tion primivitives (ProMP) and the approach we used to detect obstacles. 3.1 Probabilstic Model of Trajectories Before any predictions could be made, we first needed a model of how the subjects gait looks like. Several rep- etitions of the subjects gait had to be recorded in order to obtain this model. We had the subject wear the Xsens MC suite and walk on a treadmill at a constant velocity (0.8 m/s). After 20 steps we could than process the data and learn the model. 3.1.1 Data aqusition The human gait is a periodic movement that consist of two main phases, the stance and the swing phase [11]. We focused on the latter, because only in this one the leg moves in space. This means that we had to seperate the recorded gait cycle into this two phases. Because the foot only moves during the swing phase we observed when the velocity of the foot in the sagital plane became pos- itive. This moment indicated the end of the stance and the begining of the swing phase. The exact threshold was set empiricaly so that the small variations in velocity and the noise did not have an effect. During the learning of the model all three dimension and the time were recorded while the subject walked on a treadmill. After we pro- cessed the recorded data to extract the trajectories of the swing phases of all the gait cycles. Using these trajecto- ries we could then train the probabilistic model. 3.1.2 Encoding recorded trajectories To keep the amount of parameters needed to represent trajectories as low as possible, ProMPs uses a basis func- tion representation approach. To better understand the formulation, let us take a look at a simple example where we describe a point in time a t using this method. Let t 2R 1 J denote a basis function vector containing val- ues of J basis functions at time t. Variable w2 R J 1 represents a J-dimensional feature vector that encodes weights for each of the J basis functions. With w and t defined, a point at time t can be approximated as a t = t w = 1;t J;t w 1 w J T : This concept can be applied to multi-dimensional states by using block diagonal matrices. Let‘s assume that our variablea t now has D dimensionsa t = a 1;t a D;t T . In this case the basis function vector becomes a block di- agonal matrix t 2 R D JD and the weight vector w becomes a concatenation of the weight vectors of each di- mensionw2R JD 1 . Variablea t is now approximated as a t = t w = 2 6 4 t 0 . . . . . . . . . 0 t 3 7 5 w 1 w i w D T ; where t = 1;t 2;t J;t and w i = w 1;i w 2;i w J;i T : Using the same idea we can approximate a sequence of T states denoted by = y 1:T where = 1:T w (1) with 1:T = 1 t T T 2 R TD JD ; where the vectorw and the matrix t are the same as be- fore. In Figure 2), you can see 1 dimension of a trajectory and its approximation using the described method. In our application we used Gaussian basis functions which are often used for point to point movements. To approximate the trajectories in the previously de- scribed manner, the weights for each trajectory need to be calculated. For the i-th trajectory i the corresponding weight vector w i can be estimated using a simple least squares estimate. In our application the ordinary least square (OLS) method was used w i = T 1:T 1:T + I 1 T 1:t i ; (2) where represents a regularization parameter used to avoid numerical singularities. Its value should be small, in our case we used = 10 6 . 3.1.3 Creating the probabilistic model When the weight vectors of all trajectories are calculated, we assume their values to be normally distributed, i.e., p(w) = N (wj w ; w ). The mean w and the covari- ance matrix w can be estimated with sample mean and sample covariance of thew i vectors. 112 0 1 basis functions weights x = 0 1 movement phase data approximation movement phase Figure 2: Approximating a sequence of states using 5 basis functions and their coresponding weights. With the function approximation (1) and the weight vectors w i defined, we can define a probabilistic model for trajectories as p( jw) = T Y t=1 N (y t j t w; y ) =N (y 1:t j 1:T w; y ): This model describes the probability of observing trajec- tory , given the weight vectorw that is given as a linear basis functiony 1:t = 1:T w + y;1:T : The parameter y represents independent and identically distributed (i.i.d.) Gaussian noise in the trajectoriesy t = t w + y , where y N ( y j0; y ): On the top pane of Figure 3) you can see 1 dimension of several recordings of the leg swing trajectory. The bot- tom pane shows the model that we calculated from these recordings. The thick line represents the mean of the model and the shaded area the 1- standard deviation. The movement phase denotes the normalized time. 3.2 Computing Predictions from Observations Statistical theory tells us that we can model predictions as computing the conditional probability. First we need to define the probability distribution over the trajecto- ries , which can be computed by marginalizing out the weight vectorw. In the case of Gaussian distribution the marginal can be computed in closed form as p( ) = Z p( jw)p(w)dw = Z N (y 1:T j 1:T w; y )N (wj w ; w )dw =N (y 1:t j 1:T w; 1:T w T 1:T + y ): (3) What we get is a multivariate Gaussian distribution, the conditional probability of which we can compute in closed form. 0 1 movement phase 0 1 Figure 3: Trajectory distribution model calculated from several different trajectories. This model represents only one degree of freedom. When we receive a previously unseen point a , we can predict the most likely path of the foot (parametrized through and ) by conditioning the observed state over the weight vectors. Say that we observed a sequence of statesy t1 toy tM at m=1, 2,..., M-different time points. We declare as a concatenation of the observed states y tm and as the concatenation of the basis function matrices for the observed time points. With the observed trajectories encoded as previously described, we can obtain a conditioned distributionp(w j ) over the weight vectorsw as p(w j )/N ( j w ; 0 )p(w) :=N (w j wj ; wj ): We can compute the mean wj and the covariance matrix wj as wj = w + w T L( w ) and wj = w w T L w where L = 0 + w T 1 : With the feature mean wj and covariance matrix wj obtained, we can now use this conditional distribution to calculate the distribution over the trajectoriesp( ) using (3) p( ) =N (~ y 1:T j 1:T wj ; 1:T wj T 1:T + y ); where the predicted sequence of states ~ y 1:T is represented by the product 1:T wj . In Figure 4) a prediction is cal- culated through conditioning with one and two observed states. 3.3 Obstacle Detection For detecting obstacles the Intel RealSense D435 depth camera was used. Alongside RGB video it returns a point 113 movement phase Figure 4: A demonstration of predicting a path through condi- tioning on observed states. On the top pane only one point was observed, so the predicted path is very similar to the model mean. The dark blue area represents a1 deviation from the predicted path. On the bottom pane, two points were observed so the quality of the prediction increases. cloud - a matrix of distances from the camera to the ob- jects infront. To simplify the object detection we splited the visible 3D space of the camera in small cubes, i.e. voxels. We then used the point cloud to fill in the cor- responding voxels. Using the data from the three mark- ers above the camera, this output gets transformed to the world coordinate system. This approach also gives us a simpler and computationaly less demanding visualization of the world. Because the location of the treadmill was known, we knew which voxels represent it. This way, when voxels above the treadmill were observed, we knew there was an obstacle in the way. This computation is all done on the main computer to which also the Xsens computer streams its foot place- ment prediction alongside the current position of the sub- jects feet. This data is already in the world coordinate frame. With each iteration we checked if the predicted foot placement coincides with any of the obstacle voxels and if they did, we identified this as a potential collision. 4 Discussion The presented system was able to predict the foot place- ment of both left and right foot of a subject walking on a treadmill. When an obstacle was presented, the system accurately detected it and sent a warning if the predicted foot placement was coinciding with it. The main com- puter returned a visual feedback where the output of the camera, the current foot positions and the predicted foot placements were visible. A system like this could eventu- aly be implemented in a assistive exoskeleton, that would help the user avoid tripping by extending or shortening their step. The prediction process does not only return the final position of the foot, but also the whole path. This is why it could be used by itself for example in a gait analysis study, where certain events could be triggered in advance based on the calculated prediction. The downside of the system is that because the pre- diction model is based on the previous recordings of the subjects gait, the system will only work accurately for this specific movement. Changing the subject or even the velocity of the treadmill will result in much worse predic- tions. Implementing a phase estimation like in [9] could help with the changes in velocity. 5 Aknowledgements This work was supported by the European Union’s Hori- zon 2020 through the SPEXOR project (contract no. 687- 662); AnDy project (contract nr. 731540); and by the Slovenian Research Agency (research core funding no.P2- 0076). References [1] W. R. Berg, H. M. Alessio, E. M. Mills, and T. O. N. G. Chen, “Circumstances and consequences of falls in inde- pendent community- dwelling older adults,” pp. 261–268, 1997. [2] U. N. P. F. U. Internationa, , and HelpAge, Ageing in the Twenty-First Century: A Celebration and A Challenge, 2012. [3] A. M. Schillings, J. Duysens, and S. Maartenskliniek, “Muscular Responses and Movement Strategies During Stumbling Over Obstacles,” pp. 2093–2102, 2000. [4] B. M. H. V . Wezel and J. Duysens, “Mechanically induced stumbling during human treadmill walking,” vol. 67, 1996. [5] M. Arvin, M. J. Hoozemans, M. Pijnappels, J. Duysens, S. M. Verschueren, and J. H. van Die¨ en, “Where to step? Contributions of stance leg muscle spindle afference to planning of mediolateral foot placement for balance con- trol in young and old adults,” 2018. [6] C. F. Lei Zhang, “Predicting foot placement for balance through a simple model with swing leg dynamics, 2018.” [7] A. Paraschos, C. Daniel, J. Peters, and G. Neumann, “Probabilistic Movement Primitives,” Advances in Neural Information Processing Systems 26, pp. 2616–2624, 2013. [Online]. Available: http://papers.nips.cc/paper/5177- probabilistic-movement-primitives.pdf [8] ——, “Using probabilistic movement primitives in robotics,” pp. 529–551, 2018. [9] G. Maeda, M. Ewerton, G. Neumann, R. Lioutikov, and J. Peters, “Phase estimation for fast action recognition and trajectory generation in human–robot collaboration,” pp. 1579–1594, 2017. [10] E. Rueckert, J. Camernik, J. Peters, and J. Babic, “Prob- abilistic Movement Models Show that Postural Control Precedes and Predicts V olitional Motor Control,” Scien- tific Reports, vol. 6, no. February, pp. 1–12, 2016. [11] A. Alvarez-alvarez, “Linguistic Description of the Human Gait Quality,” vol. 26, no. 1, pp. 13–23, 2013.