Zbornik 20. mednarodne multikonference INFORMACIJSKA DRUŽBA - IS 2017 Zvezek H Proceedings of the 20th International Multiconference INFORMATION SOCIETY - IS 2017 Volume H Robotika Robotics Uredila / Edited by Andrej Gams, Aleš Ude http://is.ijs.si 9.–13. oktober 2017 / 9–13 October 2017 Ljubljana, Slovenia Zbornik 20. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2017 Zvezek H Proceedings of the 20th International Multiconference INFORMATION SOCIETY – IS 2017 Volume H Robotika Robotics Uredila / Edited by Andrej Gams, Aleš Ude http://is.ijs.si 9. - 13. oktober 2017 / 9th – 13th October 2017 Ljubljana, Slovenia Urednika: Andrej Gams Odsek za avtomatiko, biokibernetiko in robotiko Institut »Jožef Stefan«, Ljubljana Aleš Ude Odsek za avtomatiko, biokibernetiko in robotiko Institut »Jožef Stefan«, Ljubljana Založnik: Institut »Jožef Stefan«, Ljubljana Priprava zbornika: Mitja Lasič, Vesna Lasič, Lana Zemljak Oblikovanje naslovnice: Vesna Lasič Dostop do e-publikacije: http://library.ijs.si/Stacks/Proceedings/InformationSociety Ljubljana, oktober 2017 Kataložni zapis o publikaciji (CIP) pripravili v Narodni in univerzitetni knjižnici v Ljubljani COBISS.SI-ID=30874151 ISBN 978-961-264-119-1 (pdf) PREDGOVOR MULTIKONFERENCI INFORMACIJSKA DRUŽBA 2017 Multikonferenca Informacijska družba (http://is.ijs.si) je z dvajseto zaporedno prireditvijo osrednji srednjeevropski dogodek na področju informacijske družbe, računalništva in informatike. Letošnja prireditev je ponovno na več lokacijah, osrednji dogodki pa so na Institutu »Jožef Stefan«. Informacijska družba, znanje in umetna inteligenca so spet na razpotju tako same zase kot glede vpliva na človeški razvoj. Se bo eksponentna rast elektronike po Moorovem zakonu nadaljevala ali stagnirala? Bo umetna inteligenca nadaljevala svoj neverjetni razvoj in premagovala ljudi na čedalje več področjih in s tem omogočila razcvet civilizacije, ali pa bo eksponentna rast prebivalstva zlasti v Afriki povzročila zadušitev rasti? Čedalje več pokazateljev kaže v oba ekstrema – da prehajamo v naslednje civilizacijsko obdobje, hkrati pa so planetarni konflikti sodobne družbe čedalje težje obvladljivi. Letos smo v multikonferenco povezali dvanajst odličnih neodvisnih konferenc. Predstavljenih bo okoli 200 predstavitev, povzetkov in referatov v okviru samostojnih konferenc in delavnic. Prireditev bodo spremljale okrogle mize in razprave ter posebni dogodki, kot je svečana podelitev nagrad. Izbrani prispevki bodo izšli tudi v posebni številki revije Informatica, ki se ponaša s 40-letno tradicijo odlične znanstvene revije. Odlične obletnice! Multikonferenco Informacijska družba 2017 sestavljajo naslednje samostojne konference:  Slovenska konferenca o umetni inteligenci  Soočanje z demografskimi izzivi  Kognitivna znanost  Sodelovanje, programska oprema in storitve v informacijski družbi  Izkopavanje znanja in podatkovna skladišča  Vzgoja in izobraževanje v informacijski družbi  Četrta študentska računalniška konferenca  Delavnica »EM-zdravje«  Peta mednarodna konferenca kognitonike  Mednarodna konferenca za prenos tehnologij - ITTC  Delavnica »AS-IT-IC«  Robotika Soorganizatorji in podporniki konference so različne raziskovalne institucije in združenja, med njimi tudi ACM Slovenija, SLAIS, DKZ in druga slovenska nacionalna akademija, Inženirska akademija Slovenije (IAS). V imenu organizatorjev konference se zahvaljujemo združenjem in inštitucijam, še posebej pa udeležencem za njihove dragocene prispevke in priložnost, da z nami delijo svoje izkušnje o informacijski družbi. Zahvaljujemo se tudi recenzentom za njihovo pomoč pri recenziranju. V 2017 bomo petič podelili nagrado za življenjske dosežke v čast Donalda Michija in Alana Turinga. Nagrado Michie-Turing za izjemen življenjski prispevek k razvoju in promociji informacijske družbe bo prejel prof. dr. Marjan Krisper. Priznanje za dosežek leta bo pripadlo prof. dr. Andreju Brodniku. Že šestič podeljujemo nagradi »informacijska limona« in »informacijska jagoda« za najbolj (ne)uspešne poteze v zvezi z informacijsko družbo. Limono je dobilo padanje slovenskih sredstev za akademsko znanost, tako da smo sedaj tretji najslabši po tem kriteriju v Evropi, jagodo pa »e-recept«. Čestitke nagrajencem! Bojan Orel, predsednik programskega odbora Matjaž Gams, predsednik organizacijskega odbora i FOREWORD - INFORMATION SOCIETY 2017 In its 20th year, the Information Society Multiconference (http://is.ijs.si) remains one of the leading conferences in Central Europe devoted to information society, computer science and informatics. In 2017 it is organized at various locations, with the main events at the Jožef Stefan Institute. The pace of progress of information society, knowledge and artificial intelligence is speeding up, and it seems we are again at a turning point. Will the progress of electronics continue according to the Moore’s law or will it start stagnating? Will AI continue to outperform humans at more and more activities and in this way enable the predicted unseen human progress, or will the growth of human population in particular in Africa cause global decline? Both extremes seem more and more likely – fantastic human progress and planetary decline caused by humans destroying our environment and each other. The Multiconference is running in parallel sessions with 200 presentations of scientific papers at twelve conferences, round tables, workshops and award ceremonies. Selected papers will be published in the Informatica journal, which has 40 years of tradition of excellent research publication. These are remarkable achievements. The Information Society 2017 Multiconference consists of the following conferences:  Slovenian Conference on Artificial Intelligence  Facing Demographic Challenges  Cognitive Science  Collaboration, Software and Services in Information Society  Data Mining and Data Warehouses  Education in Information Society  4th Student Computer Science Research Conference  Workshop Electronic and Mobile Health  5th International Conference on Cognitonics  International Conference of Transfer of Technologies - ITTC  Workshop »AC-IT-IC«  Robotics The Multiconference is co-organized and supported by several major research institutions and societies, among them ACM Slovenia, i.e. the Slovenian chapter of the ACM, SLAIS, DKZ and the second national engineering academy, the Slovenian Engineering Academy. In the name of the conference organizers we thank all the societies and institutions, and particularly all the participants for their valuable contribution and their interest in this event, and the reviewers for their thorough reviews. For the fifth year, the award for life-long outstanding contributions will be delivered in memory of Donald Michie and Alan Turing. The Michie-Turing award will be given to Prof. Marjan Krisper for his life-long outstanding contribution to the development and promotion of information society in our country. In addition, an award for current achievements will be given to Prof. Andrej Brodnik. The information lemon goes to national funding of the academic science, which degrades Slovenia to the third worst position in Europe. The information strawberry is awarded for the medical e-recipe project. Congratulations! Bojan Orel, Programme Committee Chair Matjaž Gams, Organizing Committee Chair ii KONFERENČNI ODBORI CONFERENCE COMMITTEES International Programme Committee Organizing Committee Vladimir Bajic, South Africa Matjaž Gams, chair Heiner Benking, Germany Mitja Luštrek Se Woo Cheon, South Korea Lana Zemljak Howie Firth, UK Vesna Koricki Olga Fomichova, Russia Mitja Lasič Vladimir Fomichov, Russia Robert Blatnik Vesna Hljuz Dobric, Croatia Aleš Tavčar Alfred Inselberg, Israel Blaž Mahnič Jay Liebowitz, USA Jure Šorn Huan Liu, Singapore Mario Konecki Henz Martin, Germany Marcin Paprzycki, USA Karl Pribram, USA Claude Sammut, Australia Jiri Wiedermann, Czech Republic Xindong Wu, USA Yiming Ye, USA Ning Zhong, USA Wray Buntine, Australia Bezalel Gavish, USA Gal A. Kaminka, Israel Mike Bain, Australia Michela Milano, Italy Derong Liu, Chicago, USA Toby Walsh, Australia Programme Committee Bojan Orel, chair Mitja Luštrek Niko Schlamberger Franc Solina, co-chair Marko Grobelnik Stanko Strmčnik Viljan Mahnič, co-chair Nikola Guid Jurij Šilc Cene Bavec, co-chair Marjan Heričko Jurij Tasič Tomaž Kalin, co-chair Borka Jerman Blažič Džonova Denis Trček Jozsef Györkös, co-chair Gorazd Kandus Andrej Ule Tadej Bajd Urban Kordeš Tanja Urbančič Jaroslav Berce Marjan Krisper Boštjan Vilfan Mojca Bernik Andrej Kuščer Baldomir Zajc Marko Bohanec Jadran Lenarčič Blaž Zupan Ivan Bratko Borut Likar Boris Žemva Andrej Brodnik Janez Malačič Leon Žlajpah Dušan Caf Olga Markič Saša Divjak Dunja Mladenič Tomaž Erjavec Franc Novak Bogdan Filipič Vladislav Rajkovič Andrej Gams Grega Repovš Matjaž Gams Ivan Rozman iii Invited lecture AN UPDATE FROM THE AI & MUSIC FRONT Gerhard Widmer Institute for Computational Perception Johannes Kepler University Linz (JKU), and Austrian Research Institute for Artificial Intelligence (OFAI), Vienna Abstract Much of current research in Artificial Intelligence and Music, and particularly in the field of Music Information Retrieval (MIR), focuses on algorithms that interpret musical signals and recognize musically relevant objects and patterns at various levels -- from notes to beats and rhythm, to melodic and harmonic patterns and higher-level segment structure --, with the goal of supporting novel applications in the digital music world. This presentation will give the audience a glimpse of what musically "intelligent" systems can currently do with music, and what this is good for. However, we will also find that while some of these capabilities are quite impressive, they are still far from (and do not require) a deeper "understanding" of music. An ongoing project will be presented that aims to take AI & music research a bit closer to the "essence" of music, going beyond surface features and focusing on the expressive aspects of music, and how these are communicated in music. This raises a number of new research challenges for the field of AI and Music (discussed in much more detail in [Widmer, 2016]). As a first step, we will look at recent work on computational models of expressive music performance, and will show some examples of the state of the art (including the result of a recent musical 'Turing test'). References Widmer, G. (2016). Getting Closer to the Essence of Music: The Con Espressione Manifesto. ACM Transactions on Intelligent Systems and Technology 8(2), Article 19. iv KAZALO / TABLE OF CONTENTS Robotika / Robotics .................................................................................................................................................... 1 PREDGOVOR / FOREWORD ................................................................................................................................. 3 PROGRAMSKI ODBORI / PROGRAMME COMMITTEES ..................................................................................... 5 Compliant Bimanual Actions through Learning of Primitives / Gams Andrej, Ude Aleš ......................................... 7 Collaborative Tasks Synthesis through a Hierarchical Database / Deniša Miha, Ude Aleš ................................. 11 Active Reconfiguration of Software and Hardware in a Robotic Workcell / Bevec Robert, Gašpar Timotej, Ridge Barry, Bem Martin, Kovač Igor, Gosar Žiga, Ude Aleš .......................................................................... 15 SMACHA: An API for Rapid State Machine Assembly / Ridge Barry .................................................................. 19 Extending the Workspace of Pseudo-Linear Variable-Lever Variable Stiffness Actuator / Dežman Miha, Gams Andrej ..................................................................................................................................................... 23 Soft Humanoid Robots / Hosada Koh................................................................................................................... 27 Semantic Reasoning under Realistic Conditions / Ramirez-Amaro Karinne, Cheng Gordon .............................. 28 Integrating Multi-modal Tactile Signals to a Compliant Control to Improve physical HRI / Dean-Leon Emmanuel, Cheng Gordon ............................................................................................................................... 29 Event-Driven Systems for Efficient Reactive Control with Large Scale Robot Skin / Berger Florian, Dean- Leon Emmanuel, Cheng Gordon ...................................................................................................................... 30 Indeks avtorjev / Author index ................................................................................................................................ 31 v vi Zbornik 20. mednarodne multikonference INFORMACIJSKA DRUŽBA – IS 2017 Zvezek H Proceedings of the 20th International Multiconference INFORMATION SOCIETY – IS 2017 Volume H Robotika Robotics Uredila / Edited by Andrej Gams, Aleš Ude http://is.ijs.si 13. oktober 2017 / 13th October 2017 Ljubljana, Slovenia 1 2 PREDGOVOR Pod okriljem multikonference »Informacijska družba« po krajšem premoru zopet organiziramo tudi konferenco Robotika, s katero nadaljujemo tradicijo raziskovalne robotike v Sloveniji. Robotika je v vzponu in čeprav jo mnogi še zmeraj dojemajo kot znanstveno fantastiko, je tudi uporabniška robotika že nekaj časa nekaj povsem realnega in oprijemljivega, kmalu pa bo tudi že nekaj običajnega. Robotika je tudi skorajda vseprisotna. Brez robotskih manipulatorjev si ne znamo več predstavljati sodobnih industrijskih procesov. Ne presenečajo niti kirurški roboti ali servisni mobilni roboti, ki dostavljajo pakete in hrano ter čistijo in stražijo javno infrastrukturo. Domišljija in pa želje ljudi ne poznajo mej, zato se raziskovalna robotika trudi z razvojem velikih večnamenskih robotskih hišnih pomočnikov. Pri razvoju tako kompleksnih in avtonomnih sistemov, kar nekateri ocenjujejo, da je težje kot raketna znanost, je pomembna izmenjava idej in mnenj, kar je tudi namen konference Robotika. V zborniku so zbrani prispevki raziskovalcev Odseka za avtomatiko, biokibernetiko in robotiko na Inštitutu Jožef Stefan, veseli pa smo, da imamo letos prispevke s svetovno priznane Tehniške Univerze v Muenchnu, Nemčija. Upamo, da bo izmenjava idej in raziskovalnih rezultatov vodila v nadaljnje skupne podvige, ki bodo še naprej pomagali soustvarjati trende raziskovalne robotike. Andrej Gams in Aleš Ude 3 FOREWORD Robotics conference in the scope of the Information Society is after a short break again a part of the multiconference, and continues the rich tradition of research robotics in Slovenia. Robotics is on the rise and even though many people still perceive it as science fiction, even consumer robotics has passed from the realm of fiction to something real, tangible. Robotics is also omnipresent. Many industrial processes today simply cannot be conceived without the use of robotic manipulators. The use of surgical and mobile service robots, which deliver packages and food and clean and guard public infrastructure is not a surprise anymore. As human imagination and wishes do not know any borders, research robotics is working hard towards the development of multipurpose, autonomous, robotic household assistants. The development of such systems, which some consider more complex than rocket science, requires cooperation between researchers and the exchange of ideas and opinions. Exchange of ideas and opinions is also the main aim and goal of the Robotics conference in the scope of the Information Society multiconference. The conference proceedings contain papers from researchers of the Department for Automatics, Biocybernetics and Robotics of Jožef Stefan Institute. We are delighted to have attracted contributions from researchers of the world-renowned Technical University of Munich, Germany. We hope that the exchange of ideas will lead to joint undertakings and will help to co-shape the trends of research robotics in the future. Andrej Gams and Aleš Ude 4 PROGRAMSKI ODBOR / PROGRAMME COMMITTEE Andrej Gams Aleš Ude 5 6 Compliant Bimanual Actions through Learning of Primitives Andrej Gams Aleš Ude Humanoid and Cognitive Robotics Lab Humanoid and Cognitive Robotics Lab Dept. of Automatics, Biocybernetics and Dept. of Automatics, Biocybernetics and Robotics Robotics Jožef Stefan Institute Jožef Stefan Institute andrej.gams@ijs.si ales.ude@ijs.si ABSTRACT Compliant Movement Primitives (CMP) provide the means to achieve low trajectory errors and compliant control with- out explicit dynamic models of the task. This paper ad- dresses the application of CMPs to bimanual tasks. Besides performing the task, the robots have to maintain their rel- ative posture even in the presence of external perturbations on any of the robots, not to exert force on the object they are carrying. Thus, they act compliantly in their absolute task, but remain stiff in their relative task. For compli- ant absolute behavior and stiff relative behavior we combine previously developed joint-space CMPs with the symmetric control approach. We further augment it by applying a vir- tual force vector at the end-effector, calculated through the Figure 1: Bimanual robot performing a task while measured external joint torques on the perturbed robot, to being perturbed by a human. increase bimanual compliance. Experiments with two Kuka LWR-4 robots in a bimanual setting show applicability of the system. plied for the repetition of the exact same action. Such an approach was utilized in [5] and termed Compliant Move- ment Primitives (CMPs). It is applicable to robots with 1. INTRODUCTION active torque control. In this paper we show how we can ex- Robots were considered dangerous to humans and objects in tend the CMP framework to make it applicable for bimanual their workspace and were thus confined to cages [7]. This is task. For bimanual operation, we need to maintain the rela- due to high stiffness and position control used to accomplish tive task. We implement this by integrating the symmetric accurate execution of their given tasks. However, the notion controller into the framework. For increased compliance, of collaborative robotics, where both the human and robot which allows safe physical interaction with humans, we also share their workspace to accomplish a common task [8], has utilize virtual force translation, where we copy the pertur- gone beyond that. Collaborative robotics applications go bation to the robot from one robotic arm to the other. The beyond the factory work-floor to everyday human environ- experimental system used is presented in Fig. 1. ments, including bimanual and humanoid robots. Safety of the human is the primary task in shared envi- 1.1 Related Work ronments. This can be ensured through the compliance of Many papers have dealt with the topics of compliant control. the robot. Compliance can be active, originating from con- It commonly relies on explicit dynamics of the robot and tact detection with an artificial tactile skin [11]. On the the task [13]. Compliant movement primitives, which repre- other hand, elastic elements can introduce passive compli- sent the basic background of this paper, mitigate this need ance. Passive compliance can also be actively changed using through learning of required torques. Similar approaches variable stiffness actuators [1]. Appropriate active torque that rely on task-specific models have been presented. [3] strategies, relying on comparing the actual torques and the used tactile sensors to determine the force of contact with required theoretical torques [9] have also been used to imple- the environment on the iCub robot, and then calculate the ment active compliance. However, such methods require the joint-torques from the measured arm pose. The calculated correct dynamic model of both the robot and of the task. joint torques were used in a feed-forward manner in con- Such models of task dynamics are often often difficult to trol. In an analogous manner in [15] the authors recorded derive. joint torques along the kinematic trajectory and then used as the feed-forward signal for increased accuracy in the next To bypass the need to develop dynamical models, one can execution of the same in-contact task. learn the specific required torques for the specific task with learning by demonstration (LbD). Learned torques are ap- Different approaches for bimanual control of robots have also 7 been presented, with a basic separation in being either asym- a combination of radial basis functions (RBFs). We again metric or symmetric control. While the former controls each refer the reader to [5] for details. robot independently, the latter considers both robots as a single system. For example, in [6] an asymmetric control CMPs operate in joint space. For bimanual tasks, the rela- scheme using motion primitives is described. The robots are tion of the robots is in task space. coupled through learned feed-forward signals. However, the robots themselves are completely stiff. A dynamical system 3. SYMMETRIC ROBOT CONTROL that combines single or bimanual robot operation based on In this paper we provide a reduced description for the control dynamical systems was presented in [14]. The authors use a of a bimanual system. The absolute coordinates describe the virtual object to define the motion of robotic arms. position and orientation of a common coordinate frame (CF) of the robots in reference to the inertial CF. The relative Asymmetric system, on the other hand, can describe a co- ones (6 DOF) describe the position and orientation of one operative operational space. Thus it allows the user to de- robot end-effector relative to the other. termine the relative and the absolute tasks. This also allows to define geometrical variables at the position/orientation The Jacobian matrix of a bimanual system includes both level [4]. An example of such is presented in [12], showing the absolute and the relative parts. a human-robot cooperation scheme for bimanual robots. It is based on separately defining the gains for absolute and J J = abs . (4) relative motion. However, trajectory tracking errors will in- Jrel crease considerably when the absolute gains are set low. We can iteratively calculate the inverse kinematics using 2. COMPLIANT MOVEMENT PRIMITIVES ˙ ~ q = J† (~ v Compliant movement primitives rely on the impedance con- d + K~ e) , (5) troller, such as the one used in the Kuka LWR robot [2], where J† is the Moore-Penrose pseudo-inverse of the Jaco- and add feed-forward torques ~ τ T f f . Thus, the low-level robot bian matrix from (4). In (5) ˙ ~ q = ˙ ~ q T ˙ is the vector 1 ~ q T 2 controller is given by of angular velocities, ~ e = ~ e T T is the vector of task abs ~ e T rel ~ τ T u = Kq (~ qd − ~ q ) + Dq( ˙ ~ qd − ˙ ~q) + ~ fdynamic(~ q, ˙ ~ q, ¨ ~ q ) + ~ τff . (1) space errors, ~ vd = ~ v T are the desired task space absd ~ v T reld velocities and K is a 12 × 12 diagonal gain matrix. The Here ~ τu is the control torque vector, Kq is a diagonal joint- absolute error is given by stiffness matrix, ~ qd and ~ q are vectors of the desired and mea- sured joint positions, respectively, D  −  q is a diagonal damping ~ p w absd ~ p w abs matrix, ~˙ q 1 d and ~ ˙ q are the desired and measured vectors of ~ e  (S (~ n w  abs = abs) ~ n w absd + S (~ s w abs) ~ s w absd+ (6) joint velocities, respectively, and ~ f  2  dynamic(~ q, ˙ ~ q, ¨ ~ q) represents the robot dynamics and the non-linearities occurring in the +S (~ a w abs) ~ a w absd) robot. For the relative coordinates we have If the robot is made compliant by lowering the stiffness (Kq), this increases the trajectory tracking error. To compensate,   feed-forward torques ~ τ Rw − abs ~ p abs reld ~ p w r f f are added to the motor torque to 1 preserve trajectory tracking. These feed-forward torques ~ τ   f f ~ e Rw ~ n 1 ~ s 1 rel =  1 S ~ n 1 rel reld + S ~ s 1 rel reld+  (7) are usually calculated from an explicit dynamical model. 2   However, for repeatable tasks, we can use previously learned +S ~ a 1 rel ~ a 1 reld torques to provide low trajectory tracking errors. That is the basic principle of the the CMP framework [5]. Desired velocities are then ˙ A CMP combines desired joint motion trajectories (joint ~ p w ~ v absd absd = (8) positions ~ q ~ ω w d(t)) and corresponding joint torque signals ~ τff (t) absd ~ h(t) = [~ q d(t), ~ τff (t)]. (2) Rw ˙ ~ p abs ~ v abs reld + S (~ ω w abs) Rw abs ~ p abs reld reld = (9) ~ ω1reld Joint positions for all degrees-of-freedom (DOF) are learned, where the subscript suffix d stands for desired, S(·) is the for example through imitation, while joint torques are recorded skew-symmetric operator and ~ n j , ~ s j , ~ a j are, respectively, i i i from a stiff execution. Because CMPs encode only task- the first, second and third column of a rotation matrix. i.e. specific torques, we gain them by subtracting the known Rj = ~ n j ~ s j ~ a j . i i i i robot’s ~ fdynamic(~ q, ˙ ~ q, ¨ ~ q ) from the actual measured torques ~ τm at robots joints To control the joint torques of a bimanual system, we can now use ~ τff = ~ τm − ~ fdynamic(~ q, ˙ ~ q, ¨ ~ q ). (3) ~ τ ˙ biman = JT Kt~ e + Dt~ e . (10) Joint positions are encoded as dynamic movement primitives Here Kt and Dt are diagonal gain matrices for stiffness and (DMPs) [10] and the corresponding torques are encoded as damping, respectively. Low values on their diagonal will 8 result in compliant behavior, resulting in poor trajectory when a perturbation occured. The robots were each car- tracking and high errors. rying a 2.5 kg load. The robots were not physically cou- pled through holding a common object, so that the relative The drawback of the controller (10) is that it changes the task conformity is clearly expressed. In this experiment we torques of both manipulators. By pushing on one robot, ad- controlled the relative position of the system, but did not ditional torques will appear in both of them to neutralize control the relative orientation. the perturbation. An unintentional collision will thus re- sult in less compliant behavior of the bimanual system. We Absolute error increase the compliancy of the bimanual system, by addi- 0.06 ] 0.04 tionally introducing a virtual force translation. [m 0.02 sb 0 e a -0.02 From measured joint torques we can calculate the end-effector -0.04 force using the virtual work theorem, which states 0 5 10 15 20 25 30 35 40 Relative error ~ τ 0.01 e = JT ~ fe. (11) 0.005 A perturbation on one robot can thus be used to estimate [m] 0 x e rel the end-effector force of that manipulator ~ f y -0.005 1e using (11). We z apply the same end-effector force through the joint torques -0.01 0 5 10 15 20 25 30 35 40 to the other robot. Thus we have Perturbation ~ f1e = ~ f2e. (12) 20 [N] 0 Only the virtual torques caused by the perturbation should F be translated to the other robot, not the complete joint -20 Fx,L Fy,L Fz,L Fx,R Fy,R Fz,R torques. These are calculated by -40 0 5 10 15 20 25 30 35 40 t [s] ∆~ τi = ~ τi expected − ~ τi measured, i = 1, 2. (13) The translated torque is thus Figure 2: Absolute error (top), relative error (mid- dle) and end-effector perturbation (calculated from  T  measured joint torques) when using the complete ~τ JT 1 J† ∆~ τ 2 2 ~ τ vft,1 controller, given by (15). vft = = . (14) ~ τ  T  vft,2 JT 2 J† ∆~ τ 1 1 τ1, RIGHT 30 4. BIMANUL CONTROL USING CMPS ff rec biman vft 20 ] If we use low gains in (10), we introduce introduce com- m 10 pliance to the system, but also high trajectory tracking er- [N τ rors. Therefore we need to add also the feed-forward torques. 0 Feed-forward torques ~ τff , which we use in (1), are now com- -10 0 5 10 15 20 25 30 35 40 posed of three components t [s] ~τ ~ τ f f,1 f f = = ~ τ ~ τ rec + ~ τbiman − ~ τvft. (15) Figure 3: The feed-forward torque on the first joint f f,2 of the right robot, and separate components when using the complete controller, given by (15). See the The pre-recorded or learned task torque ~ τrec ensures trajec- bottom plot of Fig. 2 for the perturbation. tory tracking. It is the direct output of the CMP. However, the reference joint trajectories are calculated from the task- The result of using the complete controller, given by (15) space trajectories using (5). Note that the inverse kinemat- is compliant operation in the absolute task, but maintains ics solution needs to match the posture of the robot during low errors in the relative task despite the high external per- the demonstration, which might become a problem with re- turbation forces on the one side, but also maintaining low dundant tasks. trajectory tracking errors if no perturbation is present. The results are shown in Fig. 2. The effect of including the vir- 5. EXPERIMENTAL EVALUATION tual force translation in ~ τff is seen also at seconds 18 – 20 We performed our experiments on two Kuka LWR-4 7 DOF in the bottom plot of Fig. 2. There, the right robot did not robots. We locked the rotation of the 3rd axis on both include ~ τvft. Thus, a twice higher perturbation resulted in a robots, so that our system was not redundant for the task. much lower absolute error, meaning that the system was far The system was controlled from Matlab. The robots were less compliant, when ~ τvft was not included. controlled with (1), with joint stiffness set to 25 Nm/rad for all the used joints. Figure 3 shows the complete torque τ1,RIGHT and the contri- butions of separate components of the controller for the first The task of the bimanual system was to perform a biman- joint of the right robot. It is evident that τvft,1 and τbiman,1 ual trajectory while being compliant. This means that the are similar. Thus, when perturbing the left robot, the per- trajectory tracking error was low when there were no pertur- turbation is not fighting ~ τbiman of the right robot, because bations, but the robot was compliant in the absolute task it only has to account for a much smaller relative error. A 9 Figure 4: Bimanual robotic system performing the experimental compliant bimanual task. A person was perturbing the motion. small relative error remains due to different postures of the tactile sensing from robot skin. In IEEE-RAS 15th robots that make force-vector copy inaccurate. We can also International Conference on Humanoid Robots see that when there is no perturbation, the contribution of (Humanoids), pages 690–695, Nov 2015. the bimanual symmetric torque controller and of the virtual [4] P. Chiacchio and S. Chiaverini. Kinematic control of force translation is practically 0. The plot also shows that dual-arm systems. Springer, 1998. τrec,1 is the actual learned feed-forward torque, while the [5] M. Deniša, A. Gams, A. Ude, and T. Petrič. Learning other two react to perturbations. compliant movement primitives through demonstration and statistical generalization. Figure 4 shows a series of still photos showing the bimanual IEEE/ASME Transactions on Mechatronics, PP, 2016. execution and physical interaction of the bimanual system [6] A. Gams, B. Nemec, A. J. Ijspeert, and A. Ude. with a person. Coupling movement primitives: Interaction with the environment and bimanual tasks. IEEE Transactions 6. CONCLUSION on Robotics, 30(4):816–830, Aug 2014. In this paper we have shown how one can extend the com- [7] S. Haddadin. Towards Safe Robots. Number 90 in pliant movement primitives framework to include bimanual Springer Tracts in Advanced Robotics. Springer Berlin task execution. In the experiments, the robots were not Heidelberg, 2014. physically coupled through holding a rigid object, yet we [8] S. Haddadin, A. Albu-Schaffer, and G. Hirzinger. showed that the system maintains the relative posture, but Requirements for Safe Robots: Measurements, is compliant in the absolute coordinate frame. Without per- Analysis and New Insights. The International Journal turbations, the system maintains low tracking error and that of Robotics Research, 28(11-12):1507–1527, aug 2009. is the real contribution of compliant movement primitives. [9] N. Hogan. Stable execution of contact tasks using impedance control. In Proc. IEEE Int. Conf. Robotics The presented approach can be applied to a specific, pre- and Automation (ICRA), pages 1047–1054, Raleigh, learned task. On the other hand, generalization has the NC, USA, 1987. potential to extend it for a wider region of operations. That [10] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, remains to be researched in the future. Another topic for the and S. Schaal. Dynamical Movement Primitives: future is dealing with redundant robots. When the robots Learning Attractor Models for Motor Behaviors. are redundant for the task, kinematic mapping offers nu- Neural computation, 25:328–73, 2013. merous solutions. Learning of torques for all solutions is not [11] P. Mittendorfer and G. Cheng. Humanoid multi-modal viable, as there could literally be infinite. The posture of tactile sensing modules. IEEE Trans. Robot., the robots needs to be maintained to match the posture of 27(3):401–410, 2011. learning the torques. Going beyond this with CMPs remains [12] B. Nemec, N. Likar, A. Gams, and A. Ude. Adaptive an open research question. human robot cooperation scheme for bimanual robots. In J. Lenarcic and J. Merlet, editors, Advances in Acknowledgments Robot Kinematics, pages 385–393. INRIA, 2016. This work was supported by the Slovenian Research Agency [13] J. Peters, K. Mülling, D. Kober, Jensand under grant agreement no. J2-7360. Nguyen-Tuong, and O. Krömer. Robotics Research: The 14th International Symposium (ISRR), chapter 7. REFERENCES Towards Motor Skill Learning for Robotics, pages 469–482. Springer Berlin Heidelberg, Berlin, [1] A. Bicchi and G. Tonietti. Fast and ”soft-arm” tactics Heidelberg, 2011. [robot arm design]. IEEE Robot. Automat. Mag, [14] S. S. M. Salehian, N. Figueroa, and A. Billard. 11(2):22–33, 2004. Coordinated multi-arm motion planning: Reaching for [2] R. Bischoff, J. Kurth, G. Schreiber, R. Koeppe, moving objects in the face of uncertainty. In A. Albu-Schaeffer, A. Beyer, O. Eiberger, Proceedings of Robotics: Science and Systems, S. Haddadin, A. Stemmer, G. Grunwald, and AnnArbor, Michigan, June 2016. G. Hirzinger. The KUKA-DLR Lightweight Robot [15] F. Steinmetz, A. Montebelli, and V. Kyrki. arm - a new reference platform for robotics research Simultaneous kinesthetic teaching of positional and and manufacturing. Robotics (ISR), 2010 41st force requirements for sequential in-contact tasks. In International Symposium on and 2010 6th German IEEE-RAS 15th International Conference on Conference on Robotics (ROBOTIK), pages 1–8, 2010. Humanoid Robots (Humanoids), pages 202–209, Nov [3] R. Calandra, S. Ivaldi, M. Deisenroth, and J. Peters. 2015. Learning torque control in presence of contacts using 10 Collaborative Tasks Synthesis through a Hierarchical Database Miha Deniša Aleš Ude Humanoid and Cognitive Robotics Lab. Humanoid and Cognitive Robotics Lab. Dept. of Automatics, Biocybernetics and Dept. of Automatics, Biocybernetics and Robotics Robotics Jožef Stefan Institute Jožef Stefan Institute Ljubljana, Slovenia Ljubljana, Slovenia miha.denisa@ijs.si ales.ude@ijs.si ABSTRACT Collaborative human-robot task execution is presented in this paper. Example collaborative movements are encoded in a dual hierarchical database. The primary part of the database is used for human movement recognition, while the data in the secondary database is used to synthesize appropriate collaborative robot movements. Keywords Programming by Demonstration, Motion Recognition, Col- laborative Tasks, Human-Robot Interaction 1. INTRODUCTION By moving robots into an unstructured environment, pro- Figure 1: A simple representation of a dual gramming its movements by hand becomes unfeasible. New database. The primary part, used for construction movements can alternatively be gained through program- and motion recognition, can be seen on the left. The ming by demonstration (PbD) [14, 6], where a human demon- secondary part, seen on the right, stores correspond- strates example movements. While multiple approaches can ing robot movements. be used to record human movements [16, 12, 17], we use kinesthetic guidance, where a human physically moves the 2. METHODOLOGY robot [8, 13]. The first part of the proposed approach consists of dual hi- Important aspect of using robots in a home environment erarchical database construction. A simple representation or in a small to medium enterprise, is human-robot col- of a dual hierarchical database is shown in Fig. 1. Its con- laboration. In this two step process, human intentions are struction starts by demonstrating a set of nS human-robot recognized first. Then an appropriate collaborative robot collaborative movements D. Each one contains nD state movement is synthesized and executed. Various approaches vectors, were used by several authors: Hidden Markov Models [10, D = [y1, y2, . . . , yn ], (1) D 2]; extension of DMPs, called Interaction Primitives (IPs) [1]; Probabilistic Movement Primitives [11, 7]; etc. Similar sampled at a given discrete time tD. Each state vectors, to our approach, Yamane et al. [19, 18] used a binary tree yi = [yh i , yr i ]T , (2) database to recognize and synthesize movements. includes human state vectors yh i , and corresponding robot Our presented approach is based on a dual hierarchical database state vectors yri. Human movements are used to build the of example collaborative movements. While the primary primary part of the database. Clustering is used to construct database encodes example human movements and is used multiple levels which encode all demonstrated movements for recognition, the second database encodes example corre- at different granularities. Each level includes a weighted sponding robot movements and is used for cooperative move- directed graph, i. e. a transition graph (TG). It is based ment synthesis. Previously, new movements were synthesize on demonstrated movements and represents transitions be- using a single hierarchical database [5]. An extended dual tween the nodes at that level. The secondary database en- database was used to synthesize new compliant movement codes corresponding robot movements. At each level the primitives [4]. Using a dual database for human robot col- nodes are mirrored from the primary database. This means laborative tasks was preliminary evaluated through compar- that each node in the primary database has a mirrored ison [3]. node in the secondary database, which includes robot state vectors recorded at the same time as human state vectors. 11 Figure 2: Recognition and cooperative movement synthesis overview. The further details on hierarchical database construction are omitted and the reader is referred to [5, 3]. The second part of the proposed approach uses the dual 0.4 database for human movement recognition and correspond- 0.5 ing robot movement synthesis. The process, shown in Fig. 2, can be divided into three components: human movement 0.6 ] [m recognition, which finds the most probable human nodes x 0.7 w.r.t. the current sliding window; human path prediction, 0.8 which uses transition graphs to predict a full human path; and robot movement synthesis, which uses DMPs to inter- 0.9 polate between corresponding robot nodes and generates an -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 y[m] executable participating robot trajectory. Each recognition iteration starts by updating the sliding window, which consists of last nW human state vectors ob- served in the current human movement, Figure 3: Set of demonstrated cooperative trajecto- ries seen in two dimensions Robot trajectories, de- Yo = {yo noted in red, were converted to task space for pre- n , . . . , yo }, (3) o −nW no−1, yo no sentation purposes. Human trajectories are denoted where o represents observed, and the complete number of hu- in green. man state vectors observed from the start of the movement is denoted by no. At each iteration, recognition is done by where n traversing down the levels of the primary database. Several v denotes the number of state vectors clustered in node v and m steps are done at every level l: 1) determine the considered v denotes the number of demonstrated tra- jectories passing through it. The interpolation of robot nodes at this level, i. e., the nodes who’s parents were above state vectors belonging to the most suitable robot node se- the cut-off range at the previous level; 2) build a matrix of quence, and enhanced with time stamps, is done with Dy- considered nodes, where each row of length nW is a permu- namic Movement Primitives (DMPs). The details on DMPs tation of considered nodes; 3) calculate the recognition score are omitted and the reader is referred to [15, 9]. DMPs en- for each permutation based on the TG and similarities be- sure smooth and continuous robot movements, even in the tween the observations and considered nodes; 4) determine events of sudden recognition change. the nodes with recognition score above the cut-off range, who’s children nodes will be considered as we move down a level. As we reach the last level, the permutation of nodes 3. EVALUATION with the best recognition score is determined as the most Evaluation was done on a robot system consisting of: a probable human sequence of nodes. KUKA-LWR4 robot arm, a three-fingered Barret Hand grip- per, and a passive marker based system OptiTrack. Six Next component uses these results for robot sequence pre- collaborating tasks were executed, with one demonstrator diction. It follows the most probable human sequence on the kinesthetically guiding the robot and the other executing TG w.r.t. highest weights until it reaches an end node. It the task with markers on his hand. Six human movements then mirrors it to the secondary database and predicts the and corresponding robot movements are shown in Fig. 3. most suitable robot node sequence. They are shown in two dimensions for presentational pur- poses. The robot movements were captured and encoded Last component is synthesis, which enhances the most suit- in the database in joint space. For this image they were able robot node sequence with time stamps and interpolates converted to task space. The recorded set of collaborating it. Time stamps are determined based on estimated duration movements can be divided into to two sets of three move- of nodes. Time duration t ments. Each set executes a variation of a pick and place task v of a single node v is estimated as with one of two objects. The human part of the task was to n grab up one of two object (peg or cover) and move it to one v tv = tD, (4) m of three final position for each object. He then inserted/put v 12 Recognition Synthesis 0.03 ×10-3 Prediction 2 0.02 1.5 0.015 0.02 c c c T T 1 T 0.01 0.01 0.5 0.005 0 0 0 50 100 150 200 250 50 100 150 200 250 50 100 150 200 250 Recognition Sample Recognition Sample Recognition Sample Figure 4: Computation time of components. Mean values of computation times over all 10 experiments are presented in the 3 graphs. They present computation times needed for separate components of the process. the object in/on the cylinder. The robot’s task, which was due to a satisfactory computation time. In all ten experi- holding the cylinder, was to rotate it appropriately. As de- ments the human movement was recognized and an appro- scribed in the previous section, the collaborative movements priate collaborative robot movement was synthesized. All were used to build a dual hierarchical database. robot movement were smooth and continuous. Ten experiments were performed. In each the same dual Future work involves further evaluation, including responses database was used to recognize human movement and syn- to sudden human movement changes. The approach will also thesize an appropriate collaborative robot movement. In be evaluated with a bigger set of more various demonstrated each experiment the human performed a different move- collaborative movements. Computation time needed for the ment: picking up an object and moving to one of the 6 recognition component could be further reduced to make the final positions, picking up an object and while moving it whole process perform faster. changing its mind about the final position, and moving to pick one object, but then changing his mind and picking up 5. REFERENCES the other one. [1] H. B. Amor, G. Neumann, S. Kamthe, O. Kroemer, and J. Peters. Interaction primitives for human-robot As our goal was to perform the recognition and synthesis cooperation tasks. In Robotics and Automation on-line, we first evaluated the needed computation time of (ICRA), 2014 IEEE International Conference on, the approach. Mean values of calculation times over all 10 pages 2831–2837. IEEE, 2014. experiments presented in Fig. 4 are divided over the compo- [2] H. B. Amor, D. Vogt, M. Ewerton, E. Berger, B. Jung, nents: recognition of human movement, robot node sequence and J. Peters. Learning responsive robot behavior by prediction, and robot movement synthesis. We can observe imitation. In Intelligent Robots and Systems (IROS), the majority of the computation time is used for recognition 2013 IEEE/RSJ International Conference on, pages component. With the exception of the occasional rise above 3257–3264. IEEE, 2013. the desired 30 Hz, the sum computation times of all three [3] M. Deniša, B. Nemec, and A. Ude. Cooperative components, remained approximately 0.02 s. movements through hierarchical database search. In Advanced Robotics (ICAR), 2017 18th International Images in Fig. 5 show two example executions of on-line Conference on, pages 40–46. IEEE, 2017. movement recognition and cooperative task execution. Top [4] M. Deniša, T. Petrič, T. Asfour, and A. Ude. 6 images show the human picking up the cover and placing it Synthesizing compliant reaching movements by on the object. The bottom 6 images show another example, searching a database of example trajectories. In In where the human moves the peg. We can observe the robot Proc. IEEE Int Conf on Humanoid Robots, pages executing appropriate collaborative movements. 540–543, Atlanta, GA, USA, 2013. [5] M. Deniša and A. Ude. Synthesis of new dynamic 4. CONCLUSIONS movement primitives through search in a hierarchical Human movement recognition and collaborative robot move- database of example movements. Int J Adv Robot ments synthesis was presented in this paper. A set of human- Syst., 12(137), 2015. robot collaborative movements was demonstrated and en- [6] R. Dillmann. Teaching and learning of robot tasks via coded in a dual hierarchical database. While the primary observation of human performance. Robot. Auton. part was used for human movement recognition, the sec- Syst., 47(2):109–116, 2004. ondary part was used for collaborative robot movement syn- [7] M. Ewerton, G. Neumann, R. Lioutikov, H. B. Amor, thesis. Incorporated DMP approach ensures a smooth and J. Peters, and G. Maeda. Learning multiple continuous trajectory for its execution on a robot. collaborative tasks with a mixture of interaction primitives. In Robotics and Automation (ICRA), 2015 Performed experiments showed on-line execution was viable, IEEE International Conference on, pages 1535–1542. 13 Figure 5: Images from two example executions of on-line movement recognition and cooperative task execution. The human reaching for the cover and moving it to one of the final positions, is shown in top 6 images. The robot immediately starts the suitable rotation for the cover and then moves to the appropriate end point. Bottom six images show another example execution. Here the human reaches for the peg and moves it to one of its final positions. Again, the robot executes an appropriate movement. IEEE, 2015. and C. Torras. Learning collaborative [8] M. Hersch, F. Guenter, S. Calinon, and A. Billard. impedance-based robot behaviors. In In Proc. AAAI Dynamical system modulation for robot learning via Conference on Artificial Intelligence, pages 1422–1428, kinesthetic demonstrations. IEEE Trans. Robot., Bellevue, WA, USA, 2013. 24(6):1463–1467, 2008. [14] S. Schaal. Is imitation learning the route to humanoid [9] A. J. Ijspeert, J. Nakanishi, H. Hoffmann, P. Pastor, robots? Trends Cogn. Sci., 3(6):233–242, 1999. and S. Schaal. Dynamical movement primitives: [15] S. Schaal, P. Mohajerian, and A. Ijspeert. Dynamics learning attractor models for motor behaviors. Neural systems vs. optimal control–a unifying view. Prog. Comput., 25(2):328–373, 2013. Brain Res., 165:425–445, 2007. [10] D. Lee, C. Ott, and Y. Nakamura. Mimetic [16] J. Shotton, T. Sharp, A. Kipman, A. Fitzgibbon, communication model with compliant physical contact M. Finocchio, A. Blake, M. Cook, and R. Moore. in human–humanoid interaction. The International Real-time human pose recognition in parts from single Journal of Robotics Research, 29(13):1684–1704, 2010. depth images. Commun. ACM, 56(1):116–124, 2013. [11] G. Maeda, M. Ewerton, R. Lioutikov, H. B. Amor, [17] A. Ude, C. G. Atkeson, and M. Riley. Programming J. Peters, and G. Neumann. Learning interaction for full-body movements for humanoid robots by collaborative tasks with probabilistic movement observation. Robot. Auton. Syst., 47(2):93–108, 2004. primitives. In Humanoid Robots (Humanoids), 2014 [18] K. Yamane, M. Revfi, and T. Asfour. Synthesizing 14th IEEE-RAS International Conference on, pages object receiving motions of humanoid robots with 527–534. IEEE, 2014. human motion database. In In Proc. IEEE Int Conf [12] T. B. Moeslund, A. Hilton, and V. Krüger. A survey on Robotics and Automation (ICRA), pages of advances in vision-based human motion capture 1629–1636, Karlsruhe, Germany, 2013. and analysis. Comput. Vis. Image Und., [19] Y. Yamane, K.and Yamaguchi and Y. Nakamura. 104(2):90–126, 2006. Human motion database with a binary tree and node [13] L. Rozo, S. Calinon, D. Caldwell, P. Jiménez Schlegl, transition graphs. Autonomous Robots, 30(1), 2011. 14 Active Reconfiguration of Software and Hardware in a Robotic Workcell Robert Bevec Timotej Gašpar Barry Ridge Department of Automatics, Department of Automatics, Department of Automatics, Biocybernetics, and Robotics Biocybernetics, and Robotics Biocybernetics, and Robotics Jožef Stefan Institute, Jožef Stefan Institute, Jožef Stefan Institute, Ljubljana, Slovenia Ljubljana, Slovenia Ljubljana, Slovenia robert.bevec@ijs.si timotej.gaspar@ijs.si barry.ridge@ijs.si Martin Bem Igor Kovač Aleš Ude Department of Automatics, Department of Automatics, Department of Automatics, Biocybernetics, and Robotics Biocybernetics, and Robotics Biocybernetics, and Robotics Jožef Stefan Institute, Jožef Stefan Institute, Jožef Stefan Institute, Ljubljana, Slovenia Ljubljana, Slovenia Ljubljana, Slovenia martin.bem@ijs.si igor.kovac@ijs.si ales.ude@ijs.si ABSTRACT High volume production has been a prerequisite in order to invest into automation of the manufacturing process for decades. The high cost of setup and the inflexibility of the solution meant low batch productions, often present in small and medium-sized enterprises (SMEs), were dismissed for au- tomation. In order to bring automation closer to SMEs a flex- ible solution is required that can accommodate more than one manufacturing process, allows rapid change between them and doesn’t require expertise knowledge for set up. In this paper we present a novel robotic workcell that enables ac- tive reconfiguration of software and hardware components, facilitating set up and production of several manufacturing processes on a single robotic cell. The ROS based software has been designed to be robot independent and modular. Special user interfaces have been developed for cell calibra- Figure 1: The proposed reconfigurable workcell prototype tion, programming by demonstration and set up of quality shown at Hanover fair 2017. control and part localization tasks. The proposed workcell is applicable in companies with product families, where manu- facturing processes are similar and where fast changeover is required in order to adapt to new production requirements it would be highly beneficial if robotic workcells could be quickly. Due to the emphasis put on ease of use it will also developed specifically to address such use-cases. be of interest to companies getting into automation for the first time. The main barrier to greater adoption of robot production in SMEs are the expertise needed in setting up existing solu- tions and the time for testing and fine-tuning. Since SMEs 1. INTRODUCTION usually do not have such expertise available, they avoid in- Fast changes in market demands lead to great fluctuation troducing such solutions, even when they are economically of orders down the manufacturing chain. Companies must justifiable. We can recognize that these problems are due to react quickly, efficiently, and in an economically viable way. the time costs involved in re-configuring and re-programming Robots have been successful in industrial production pro- the robot workcell for new assembly tasks, which are often cesses, when applied to repetitive tasks with long production too prohibitive to make the application of robots profitable. runs and high unit volume. However, frequent shifts in the required product type or in the number of required prod- In this paper, we present the design of a new kind of au- ucts has prevented many companies from automating their tonomous robot workcell that is attractive not only for large manufacturing processes. production lines, but also for few-of-a-kind production. We propose reducing set-up times by exploiting a number of These so-called few-of-a-kind assembly production scenarios hardware and software technologies, some of which were par- [7] are typical of SMEs. Since SMEs are the “backbone of the tially developed in prior work, and some of which are novel manufacturing industry”, e. g. providing some ∼45% of the contributions in this paper particular to the proposed work- value added by manufacturing in the European Union [8], cell design. 15 Legend The main novelty of the workcell lies in the active reconfigura- Robot control Robot Module Vision Ethernet box 1 Module Plug and Produce Connector Gripper SLRT Server Camera Ethernet, Power, tion of passive fixtures and other passive elements in the cell, Air, ROS Real-time trajectory Tool exchange communication generation and Vision feedback control Digital I/O computer Digital loops Application which can be performed by the robots installed therein. This End effector tool Interface Unit Module Dependent Periphery F/T sensor Safety reconfiguration process allows the robots to autonomously switches Micro State controller swithces Robot control Robot Module ... configure their workspace and prepare the workcell for the box 2 Gripper SLRT Server execution of new assembly tasks. Real-time trajectory Tool exchange generation and feedback control loops End effector tool F/T sensor 1.1 Related Work Application Dependent Periphery ROS Master Simulation Many surveys in recent years have followed the development Micro Computer controller Computer Robot State Publisher ... (VEROSIM) Digital Interface Robot Control Interface of reconfigurable robotic systems, both in research and in Safety Vibrating Conveyor State Unit State Machine Interface switches feeder belt swithces industry [4]. Work of Chen [2] focuses on the modular recon- figurability aspect in particular by finding optimal module Figure 2: Schematics of the workcell software and hardware assembly configurations from a given set of module compo- architecture. nents for a specific task. His subsequent work on the design of a reconfigurable robotic workcell for rapid response man- ufacturing [3] is of particular relevance with respect to the workcell proposed in this paper. However, his work involved resent a robot with its robot control SLRT server, one ad- a workcell containing hardware elements that can only be re- ditional measurement unit, and all of its tools and grippers. configured manually. Our proposed workcell focuses on intro- The “ROS Master Computer” refers to the computer in the ducing hardware elements that can be actively reconfigured system that runs the ROS core and our basic nodes. In order automatically by the system itself [5]. to access various periphery from the workcell a “Digital In- terface Unit” is used for bridging the connection from PLC to In the work of Krüger et al . [7,9], a set of methods was devel- ROS. Other typical modules include a “Vision Module” and oped to facilitate the set-up of complex automated assembly a “Simulation Computer”. The Vision Module represents a processes. The proposed set of methods included pose esti- user programmable processing unit for typical vision tasks mation and tracking of parts using a 3-D vision system, fast like quality control and part localization. The Simulation and robust robot trajectory adaptation using dynamic move- Computer offers a dynamic simulation environment provided ment primitives (DMP) [10], and ROS-based software control by the VEROSIM software package, where the production and state machine programming. In this work we build on can be planned and evaluated. Depending on requirements, these approaches and add the ability to automatically recon- the workcell design can be adapted by adding or removing figure the workcell, while adding a user interfaces to facilitate various modules. task set up. The proposed system advances beyond synthetic benchmarks and demonstrates the viability of the system on actual industrial use-case. 2.2 Simulink Real-Time Server The SLRT server, responsible for robot control, connects di- 2. RECONFIGURABLE SOFTWARE rectly to the robot controller via Ethernet. Standard robot The introduction of a robotic cell into a production line repre- controllers usually only offer basic control methods and not sents a big investment for SMEs. The high costs usually come state-of-the-art trajectory generation methods implemented from the price of the necessary hardware and the time spent on the SLRT server. Importantly this approach also makes for the integration of the robotic system into the production our system independent of the robot. Compatible robots line. One of the time-consuming aspects is the program- must enable receiving a joint stream over Ethernet and all ming of task sequences for the robots involved in the produc- it takes is a modification of the kinematics model on the tion process. The programming is normally done via on-line SLRT server in order to integrate a new robot. The SLRT programming using a robot teach pendant connected to the server also connects to some other measurement units (e.g. robot controller, or via off-line programming in a simulation force/torque sensors) that can be used for closed loop control environment. Because specific robot system knowledge is re- policies (e.g. force control). quired for both approaches, we developed a software system that would facilitate the programming of robot tasks regard- The trajectory generation algorithms that have been imple- less of the robot system. The software system is designed to mented so far cover the most common robot motion needs in be distributed, modular and offers seamless reconfiguration the context of automated assembly. These are: trapezoidal of the robot cell. The package also provides the necessary speed profile in joint space; minimum jerk for position and tools to enable simple, intuitive programming of robot tasks. minimum jerk SLERP for orientation trajectories in Carte- sian space; admittance force control [1]; joint space dynamic Our system was build using Robot Operating System (ROS) movement primitives for free-form movements [6]; Cartesian framework, where the hard real-time components were de- space dynamic movement primitives free-form movements in veloped using a Matlab Simulink Real-Time (SLRT) server, Cartesian space [10]. since ROS in its current form does not provide any form of hard real-time implementation. This is a crucial requirement for reliable and accurate robot control. 2.3 ROS Software Package To allow the robot workcell to be accessed, controlled and 2.1 System Architecture Overview calibrated within the ROS environment, various ROS nodes Elements of the software architecture of a typical workcell have been developed to offer an interface to the SLRT server design can be seen in Figure 2. The “Robot Module”s, rep- and other modules in the workcell. 16 SLRT State Publisher to read joint positions, velocities, pay- Reconfigurable Frame made of rectangular steel beams that load, tool information, force/torque sensor data from the are connected via the BoxJoint patented modular frame cou- SLRT server and publish it via ROS topics using standard pling technology. The advantage of this technology is that a ROS messages. workcell frame can be easily configured into a large variety of shapes. Action Servers handle communication to the SLRT server and are used to trigger robot motion and monitor the progress Tool Exchange System for the robots to equip themselves of the trajectory using actionlib. Each trajectory generation with different grippers needed for different assembly tasks. algorithm that is implemented on the SLRT server is offered Tools are introduced into the workcell on trolleys that con- as a separate action server with its own goal, feedback and nect to the P&P connector. If reconfiguration of the cell is result messages. The low level logic of all of the action servers needed to assemble a different workpiece, a new trolley with ensures that only one motion can be executed at a time. different end-effectors can be introduced into the workcell. ROS Services for handling short duration tasks such as chang- Passive Sensor-less Reconfigurable Fixtures designed in a Stew- ing the state of digital outputs. Our ROS package includes art platform-esque configuration with six legs, named “hexa- services for changing the robot mode from position control pod”. These fixtures can be actively reconfigured by the to gravity compensation mode, triggering direct joint control robot arms on demand by connecting a robot to a fixture on the SLRT server and setting digital outputs on the robot via the tool exchange system, releasing the fixture brakes, controller. manipulating the fixture into the desired pose, re-applying the brakes, and disconnecting the robot from the fixture. Database for keeping track of the workcell state at all times, be it during operation or downtime. We followed a common Passive Sensor-less Linear Unit, on which the robot is mounted, approach with wide support in the community and imple- to enlarge the work area of the robot within the work cell. mented a MongoDB database for persistent storage. The robot is used to propel itself along the linear axis by con- necting the end-effector to the frame and moving the base to Robot Capture Program for capturing and storing various a new position. Compared to conventional actuated solutions robot related configurations in the database. It is commonly that are much more expensive, this approach is appropriate used in conjunction with the kinesthetic guiding of the robot, for applications where the need to move the robot is relatively where the programmer of the robot workcell can freely move infrequent. the robot in its workspace and then save the points of inter- est. The functionality is commonly used for calibrating the 4. USE CASE EVALUATION workcell state (reconfigurable fixture positions, tool pick-up In order to evaluate our proposed robotic cell, we imple- slots) and for saving pick-and-place poses of the robot assem- mented a real industrial use-case. The industrial partner is bly task. involved in automotive light production, where the demand for different lights can vary substantially in a single year. Programming by Demonstration as an intuitive method for The total production of each light housing is typically be- teaching the robot how to move to either points in Cartesian tween 100,000-300,000 units per item. However, these lights or joint space or over whole trajectories. This method has are not assembled in one batch. Following the just-in-time increased in popularity in recent years and more robot manu- production paradigm, a switch from production of one auto- facturers are starting to implement the functionality on their motive light type to another is often necessary. The company robots. uses product specific assembly devices, which are stored in a warehouse, when not in use. The devices must be stored to Adaptation of Learned Trajectories by using admittance con- produce spare parts for at least the next five years. Produc- trol. The displacement due to the force error is used as a tion of spare parts in particular is a low quantity production correcting offset to our DMP encoded trajectory, when the scenario and usually occurs only a few times per year. Each learned trajectory is not ideal or optimal [1]. changeover of production lasts several hours and presents a significant cost. It would therefore be extremely useful for suppliers to have a single robot cell available which is capa- 3. RECONFIGURABLE HARDWARE ble of assembling many different types of lights, while also The proposed robotic workcell is in large part constructed of being rapidly reconfigurable for alternating production sce- modular hardware that allows for fast and easy reconfigura- narios. tion; from the structural frame to the fixtures, end-effectors, tool exchange system, P&P connectors, and other peripheral The assembly device are currently operated by people, who devices. With this approach we make it possible to use the insert parts in the machine and check the quality after assem- proposed workcell in a wide range of industrial applications bly. Manual work and quality is highly dependent on work- and environments. Furthermore, we also make it relatively ers’ qualifications, skills and their knowledge of the assembly easy to alter its shape and purpose within those environ- process. Customers expect that the supplier company is very ments. The workcell follows the notion that automatic re- flexible in coping with changes in demand. This is why SMEs configuration is achieved using robots, which actively recon- seek to time every task carefully and look for optimizations. figure the passive elements inside the workcell. This concept A fully automated assembly procedure also implements qual- drives down the cost of the cell making it more affordable for ity control checks and integrates in the company’s business SMEs. The following technologies and solutions were used to intelligence infrastructure providing key performance indica- achieve said hardware reconfigurability. tors (KPIs). Defective products can be detected before de- 17 livery and the KPIs can be used for analysis and prevention 6. ACKNOWLEDGMENTS of defects and production optimization. This work has been funded by the Horizon 2020 ICT-FoF Innovation Action no 680431, ReconCell (A Reconfigurable robot workCell for fast set-up of automated assembly pro- cesses in SMEs) and by the GOSTOP programme, contract no C3330-16-529000, co-financed by Slovenia and the EU un- der the ERDF. 7. ADDITIONAL AUTHORS Additional authors: Žiga Gosar (Elvez d.o.o., Višnja Gora, Slovenia, email: (a) Active reconfiguration of(b) Housing insertion and part ziga.gosar@elvez.si). passive fixtures. pickup. 8. REFERENCES [1] F. J. Abu-Dakka, B. Nemec, J. A. Jørgensen, T. R. Savarimuthu, N. Krüger, and A. Ude. Adaptation of manipulation skills in physical contact with the environment to reference force profiles. Autonomous Robots, 39(2):199–217, 2015. [2] I.-M. Chen. Theory and applications of modular reconfigurable robotic systems. PhD thesis, California (c) Part assembly. (d) Finished product removal Institute of Technology, 1994. [3] I.-M. Chen. A Rapidly Reconfigurable Robotics Figure 3: Key production steps for assembly of an automotive Workcell and Its Applications for Tissue Engineering. light using active reconfiguration. https://dspace.mit.edu/handle/1721.1/3753, 2003. [4] M. Fulea, S. Popescu, E. Brad, B. Mocan, and The technologies described in this paper have been imple- M. Murar. A literature survey on reconfigurable mented for light assembly in the following way (c.f . Fig. industrial robotic work cells. Applied Mechanics and 3). Before the start of the production of a new light hous- Materials, 762:233, 2015. ing model, the reconfigurable fixtures are actively placed in [5] T. Gašpar, B. Ridge, R. Bevec, M. Bem, I. Kovač, the appropriate configuration by robots, to accommodate the A. Ude, and Ž. Gosar. Rapid hardware and software housing (main body of the headlamp) that comes directly reconfiguration in a robotic workcell. In IEEE The 18th from an injection molding machine. This step happens only International Conference on Advanced Robotics once per production of a single type of headlamp (c.f . Fig. (ICAR), pages 229–236, Hong Kong, China, 2017. 3a). In the next step, the robots equip themselves with grip- [6] A. J. Ijspeert, J. Nakanishi, and S. Schaal. Learning pers with which it will pick up assembly parts. The assembly attractor landscapes for learning motor primitives. parts are detected using a calibrated visual localization sys- Advances in Neural Information Processing Systems, tem. The robot places the light housing into the fixture and pages 1523–1530, 2002. inserts the remaining parts into the housing one by one (c.f . [7] N. Krüger, A. Ude, H. G. Petersen, B. Nemec, L.-P. Fig. 3b,3c). After the assembly the robot grasps a camera to Ellekilde, T. R. Savarimuthu, J. A. Rytz, K. Fischer, inspect the quality of the assembly. Finally, the other robot A. G. Buch, D. Kraft, W. Mustafa, E. E. Aksoy, removes the housing from the fixtures, moving it on to the J. Papon, A. Kramberger, and F. Wörgötter. next step of the production process. Technologies for the Fast Set-Up of Automated Assembly Processes. KI - Künstliche Intelligenz, 5. CONCLUSION AND FUTURE WORK 28(4):305–313, Nov. 2014. In this paper we presented a reconfigurable robotic workcell [8] European Factories of the Future Research Association. that targets the manufacturing industry with small produc- Factories of the Future: Multi-Annual Roadmap for the tion batches where changes in demand happen rapidly. The Contractual PPP under Horizon 2020. Technical developed workcell consist of both modular software com- report, Publications office of the European Union: ponents and reconfigurable hardware elements. Affordable Brussels, Belgium, 2013. passive elements are actively reconfigured via robot manip- [9] T. R. Savarimuthu, A. G. Buch, C. Schlette, N. Wantia, ulation to accommodate a different manufacturing process. J. Rossmann, D. Mart´ınez, G. Alenyá, C. Torras, To demonstrate the benefit of using such a workcell in an ac- A. Ude, B. Nemec, A. Kramberger, F. Wörgötter, E. E. tual industrial scenario, a case study has been implemented Aksoy, J. Papon, S. Haller, J. Piater, and N. Krüger. with a partner from the automotive industry. In our exper- Teaching a Robot the Semantics of Assembly Tasks. iments we demonstrated that the developed reconfigurable IEEE Transactions on Systems, Man, and Cybernetics: robot workcell provides the much needed flexibility and fast Systems, 2017. doi: 10.1109/TSMC.2016.2635479. changeover characteristics for automated assembly processes [10] A. Ude, B. Nemec, T. Petrič, and J. Morimoto. in the context of automotive lights product family. In the Orientation in cartesian space dynamic movement future, we will focus on methods for finding a workcell con- primitives. In IEEE International Conference on figuration of reconfigurable components for assembly of a new Robotics and Automation (ICRA), pages 2997–3004, product automatically, taking into account the constraints of Hong Kong, 2014. the assembly procedure and the current workcell components. 18 SMACHA: An API for Rapid State Machine Assembly Barry Ridge Humanoid and Cognitive Robotics Laboratory Department of Automatics, Biocybernetics and Robotics Jožef Stefan Institute, Ljubljana, Slovenia barry.ridge@ijs.si ABSTRACT Templates Given the burgeoning complexity and diversity of both the (Jinja2) SMACHA API hardware and software components of robotic systems, soft- ware libraries that use state machines as a basis for robot Templater SMACH control by seamlessly connecting between low-level impera- Code Generator tive task scripting and higher-level task planning have been (Python) in active development over the past decade or so. However, Parser while they provide much in terms of power and flexibility, Scripts their overall task-level simplicity can often be obfuscated at (YAML) the script-level by boilerplate code, intricate structure and lack of code reuse between state machine prototypes. To ad- dress these issues, we propose a code generation, templating Figure 1: SMACHA API overview. and meta-scripting methodology for state machine assem- bly, as well as an accompanying application programming interface (API) for the rapid, modular development of robot control programs. The API has been developed within the be generated. However, the generated code is language- ROS ecosystem to function effectively as either a front-end specific and would therefore be brittle with respect to any for concise scripting or a back-end for code generation for significant changes to the programmatic approach. visual programming systems. Its capabilities are demon- strated in experiments using the Baxter robot simulator. Here we present an application programming interface (API) named SMACHA1 that aims at distilling the task-level sim- plicity of SMACH into compact scripts in the foreground, 1. INTRODUCTION while retaining all of its power and flexibility in code tem- The Robot Operating System (ROS) has, in recent years, plates and a custom code generation engine in the back- become a popular choice of middleware for communication ground. One of the major potential advantages of SMACHA and control when designing robotic applications, and var- is that it is is designed to be both language and framework ious packages within its ecosystem have come to the fore agnostic. Although this has not yet been implemented, it as being especially useful for dictating control flow. The would be possible, for example, to design templates to gen- SMACH high-level executive [3], in particular, has proven to erate FlexBE code instead of SMACH code, or even state be an exceptionally versatile and robust task-level architec- machine code written in a language other than Python, while ture for state machine construction in ROS-based systems. maintaining the same scripting front-end. It allows for the description of nested hierarchical state ma- chines in Python in which parent container states contain 2. SMACHA API OVERVIEW child state sequences. State machines may describe lists of The SMACHA API is composed of three main components different possible outcomes and transitions are specified be- as depicted in Fig. 1: a parser, a templater and a gener- tween states that depend on the outcomes in order to spec- ator. The parser parses simple data-oriented scripts that ify the control flow. These transitions are easily remapped describe the high-level arrangement of state machines to be across different depth levels in the hierarchy. Data may be constructed into operational program code by the generator passed between states as defined by a userdata object and and templater. We refer to this concept as meta-scripting the inputs and outputs of states may be remapped to user- and it is described below in Section 2.1. The templater re- data variables in order to control the flow of data. trieves and renders code templates as required by the genera- tor in order to produce the code, and is described in Section While the ideas encapsulated by SMACH are conceptually 2.2. The generator recursively processes the parsed script simple, its usage still demands a significant degree of domain- and generates the final program code using the templater. specific expertise and prototyping time in order to define a It is described in Section 2.3. The relationship between the functional state machine for a given robot control applica- scripting and templating functionality, as well as the overall tion. Another library that builds on the functionality of recursive code generation process, is depicted in Fig. 2. SMACH named FlexBE [5] aims at addressing this by pro- viding a visual programming interface from which code may 1https://github.com/ReconCell/smacha 19 SMACHA Script Base Template ... Generated SMACH Code {% block body %} name: sm ...{{ body }}... .. template: BaxterBase {% endblock body %} ... ... ... ... states: with sm: ... ... - PICK_BLOCK: with sm_pick_block: BaxterBase Template template: StateMachine StateMachine.add('IK_PICK_BLOCK_HOVER_POSE',...) ... {% extends Base %} StateMachine.add('MOVE_TO_PICK_BLOCK_HOVER_POSE',...) states: ... ... - IK_PICK_BLOCK_HOVER_POSE: {% block body %} StateMachine.add('PICK_BLOCK', sm_pick_block,...) template: PoseToJointTrajServiceState {{ super() }} ... ... {{ body }} ... ...Rendered Container Code... ... - MOVE_TO_PICK_BLOCK_HOVER_POSE: {% endblock body %} template: MoveToJointPositionsState ... ... StateMachine Template ... {% block body %} PoseToJointTrajServiceState Template {{ body }} MoveToJointPositionsState Template ... ...Rendered State Code... ... {% block body %} ...Rendered State Code... {% block body %} ...State Template Code... ... ...State Template Code... {% endblock body %} {% endblock body %} {% endblock body %} ... ... ... Figure 2: SMACHA recursive meta-scripting, templating and code generation pipeline example. Dashed arrows show nested state template selection from the SMACHA script and the blue shaded boxes indicate the depth level in the state hierarchy. Solid arrows and green shaded boxes show recursive template rendering flow, from child state templates at bottom-left and bottom-right, to a parent container StateMachine template at bottom-centre, to its parent BaxterBase template in the middle, to the final generated SMACH code on the right. Template inheritance is indicated by the dotted arrow and orange boxes. 2.1 Meta-Scripting 1 --- # Modular SMACHA pick and place test script for the Baxter simulator. One of the core ideas behind the development of SMACHA 2 name: sm 3 template: BaxterBase 4 node_name: baxter_smach_pick_and_place_test is that state machines are essentially simple entities that can 5 outcomes: [succeeded, aborted, preempted] 6 userdata: be almost entirely described via natural language constructs, 7 hover_offset: [[0.0, 0.0, 0.15], [0.0, 0.0, 0.0, 1.0]] 8 states: perhaps augmented by some essential additional information 9 - LOAD_TABLE_MODEL: 10 template: LoadGazeboModelState necessary to describe how transitions should occur and how 11 model_name: cafe_table 12 model_path: rospkg.RosPack().get_path(’baxter_sim_examples’) + data should be passed between states. With this in mind, 13 ’/models/cafe_table/model.sdf’ 14 userdata: in order to transcribe the high-level logic of state machine 15 table_model_pose_world: Pose(position=Point(x=1.0, y=0.0, z=0.0)) 16 table_model_ref_frame: world 17 remapping: {pose: table_model_pose_world, description in as simple a manner as possible with a view 18 reference_frame: table_model_ref_frame} 19 transitions: {succeeded: LOAD_BLOCK_MODEL} towards offloading the more complex aspects to be processed 20 21 - LOAD_BLOCK_MODEL: by a code generation system working in the background 22 template: LoadGazeboModelState 23 model_name: block we selected YAML (YAML Ain’t Markup Language) as our 24 model_path: rospkg.RosPack().get_path(’baxter_sim_examples’) + 25 ’/models/block/model.urdf’ scripting front-end [2]. YAML scripts are data-oriented and 26 userdata: 27 block_model_pick_pose_world: [[0.6725, 0.1265, 0.7825], so are built around constructs such as lists and associative 28 [0.0, 0.0, 0.0, 1.0]] 29 block_model_pick_ref_frame: world 30 block_model_pick_pose: [[0.7, 0.15, -0.129], arrays that may be easily translated into corresponding ma- 31 [-0.02496, 0.99965, 0.00738, 0.00486]] 32 block_model_place_pose: [[0.75, 0.0, -0.129], chine code constructs and, more importantly for our pur- 33 [-0.02496, 0.99965, 0.00738, 0.00486]] 34 remapping: {pose: block_model_pick_pose_world, poses, can be used to represent both sequences of states 35 reference_frame: block_model_pick_ref_frame} 36 transitions: {succeeded: MOVE_TO_START_POSITION} and their individual data representations respectively. They 37 38 - MOVE_TO_START_POSITION: can also represent data hierarchies very effectively, and are 39 template: MoveToJointPositionsState 40 limb: left therefore well-suited to describing SMACH container states 41 userdata: {joint_start_positions: 42 [-0.08000, -0.99998, -1.18997, 1.94002, 0.67000, 1.03001, -0.50000]} 43 and nested state hierarchies. Thus, SMACHA scripts are remapping: {positions: joint_start_positions} 44 transitions: {succeeded: PICK_BLOCK} 45 YAML files that are used to describe how SMACHA should 46 - PICK_BLOCK: 47 script: pick_block generate SMACH code. An example of a script that was 48 remapping: {pick_pose: block_model_pick_pose, 49 hover_offset: hover_offset} written for a pick and place demonstration for the Baxter 50 transitions: {succeeded: PLACE_BLOCK} 51 simulator can be see in Listing 1. 52 - PLACE_BLOCK: 53 script: place_block 54 remapping: {place_pose: block_model_place_pose, 55 hover_offset: hover_offset} 56 transitions: {succeeded: succeeded} 2.1.1 Base Variables Listing 1: SMACHA pick and place demo script. The base of a main SMACHA script file specifies the fol- lowing variables: name (a name for the overall state ma- chine), template (the name of its base template), manifest (an optional ROS manifest name), node name (a name for 2.1.2 States its associated ROS node), outcomes (a list of its possible Each state, including the base, must specify a template from outcomes) and states (a list of its constituent states). Each which its respective code should be generated (see e.g. lines of the states in the base script may, in turn, specify similar 3, 10, 22 and 39 of Listing 1). States may be specified as lists variables of their own, as discussed in the following. specifying their transition order (see e.g. lines 8, 9, 21, 38, 46 20 and 52 of Listing 1), and may also be nested as described in The most important block for most state templates is the the SMACH documentation using appropriate combinations body block and its associated body variable, as it is where the of template and state specifications. Possible state outcomes state template should render the code necessary to add the may be specified as a list in the base state machine and in state to the parent state machine, which will either be some each container state (see e.g. line 5 of Listing 1). Possible container state or the base state machine itself. Note that state transitions may be specified as an associative array in all of the above code generation variables and code blocks each state (see e.g. lines 19, 36, 44, 50 and 56 of Listing 1). may be either removed, modified or arbitrarily customized Input and output remappings of user data may be specified within the API for particular use cases. The code insertion as an associative array in each state (see e.g. lines 17, 34, order may also be specified within the API, i.e. code may 43, 48 and 54 of Listing 1). be either prepended or appended to a variable. An example of how code generation variables work together with code 2.1.3 Modularity blocks is depicted in Fig. 2. Modularity is achieved at the scripting level by allowing use- ful subroutines wrapped in container states to be saved as 2.2.3 Template Inheritance separate YAML script files called sub-scripts which can be Jinja2 provides powerful functionality, including the abil- included in a main script as states. Examples of this can ity to extend templates via template inheritance, such that be seen in lines 46–50 and 52–56 of Listing 1, where the their constituent code blocks may be overridden or extended. sub-scripts “pick block” and ”place block” are included in SMACHA aims to incorporate as much of this functionality the main pick and place state machine script to define its as possible, thus the core templates may be overridden or sub-states. The input and output userdata keys expected by augmented by custom user-specified templates via the usual the container states in the sub-scripts may be remapped as Jinja2 template inheritance mechanisms, with some caveats. appropriate in the main script along with their state transi- This works in the usual way using the following Jinja2 vari- tions. The use of this functionality encourages low coupling ables and expressions: {% extends "" %} and high cohesion, while allowing for extremely rapid and (to inherit code blocks from the parent template specified by easily specified reuse of common patterns. ), {{ super() }} (when this appears in a block, the code from the same block in the parent template 2.2 Templating as specified by {{ extends }} will be rendered at its po- sition) and {% include "" %} (to include Code templating is implemented using the Jinja2 templat- all code from the template specified by ). ing library [1]. Core templates are provided by default to support standard SMACH states and custom templates may Regarding the aforementioned caveats, there is a behaviour be defined for particular use cases. that is specific to SMACHA that goes beyond the usual ca- pabilities of Jinja2 and that was designed as a means of deal- 2.2.1 Core Templates ing with the recursive state machine processing required by SMACHA provides default core templates for many of the this particular use case. If a state template contains blocks, SMACH states and containers, as well as for other useful but does not contain an {{ extends }} expression at the constructs. At the time of writing, the following core tem- top of a template, it is implied that the code for the blocks plates are present and functional: Base (Python script skele- will be rendered into variables and blocks with the same ton), State (contains functionality common to all states, e.g. names as the blocks in the state template as dictated by the userdata specification), StateMachine (container), Concur- SMACHA script and as defined usually either by the base rence (container), ServiceState (generic state), SimpleAc- template or container templates. In the current implementa- tionState (generic state), ReadTopicState (custom state used tion, only base templates use the {% extends %} inheritance for reading messages from ROS topics) and TF2ListenerState mechanism, whereas state and container templates use the (custom state used for reading TF2 transforms). {% include %} mechanism to inherit code from other tem- plates. This is partially illustrated in Fig. 2. 2.2.2 Code Generation Variables and Code Blocks There are a number of core code generation variables and 2.3 Code Generation code blocks present in the core templates that enable SMACHA The SMACHA code generator is a custom-designed engine to produce code in the appropriate places. In most cases, a for recursively generating state machine code based on the code block contains a variable of the same name within it scripts described in Section 2.1 and using the templates de- to indicate where code from child state templates should scribed in Section 2.2. Recursive processing was necessary be rendered into. The main code blocks are as follows: given the potentially arbitrary depth levels of state machine base header (for code that must appear near the top of the nesting that are possible under the SMACH API. The ba- program script), defs (for function definitions), class defs sic operation scheme behind the code generator is thus to (for class definitions), main def (for the main function def- iterate through the data constructs of a parsed script, eval- inition), header (for code that is to be rendered into the uate them based on their type, and determine whether they header variable the parent template), body (for code that is should be rendered as code using the appropriate templates, to be rendered into the body variable of the parent template), passed on for recursive processing, or some combination of footer (for code that is to be rendered into the footer variable both. When iteratively processing a script, data items that of the parent template), execute (for the code necessary for are encountered are either lists or associative arrays. When executing the state machine), base footer (for any code that a list is encountered, it is assumed that it is a list of states must appear near the bottom of the program script) and and is passed on for further recursive processing. When pro- main (for the code necessary to execute the main function). cessing an associative array, there are three main cases that 21 Figure 3: Pick and place (left) and stacking (right) tasks running on the Baxter simulator using SMACHA-generated code. need to be handled separately: container states, sub-script script in order to pick the block from the table, followed by states and leaf states. The recursive processing of container a ”PLACE BLOCK” state as specified by the “place block” and leaf states is partially illustrated in Fig. 2. sub-script in order to place the block at a given placement pose. The stacking experiment initialises similarly to the pick and place experiment, only in this case, two block mod- 3. EXPERIMENTS els are loaded instead of one, and the robot is tasked with For the experiments, we chose to use the Rethink Robotics stacking one on top of the other. This essentially involves Baxter robot [4] simulator which uses the Gazebo simulation two pick and place sequences, one for each block, so the system and comes equipped with extensive ROS support by “pick block” and “place block” sub-scripts used in the previ- default. Custom code templates were designed to facilitate ous experiment are reused. The results of both experiments the development of the necessary states required for the ex- are depicted in Fig. 3. periments. Two experiments were performed in total using these templates: a pick and place experiment and a block 4. CONCLUSION stacking experiment2. The first of these is a replication of the pick and place demo that comes as standard with the We have developed an API for the rapid assembly of state Baxter SDK. It was initially re-programmed from scratch in machines for modular robot control using a meta-scripting, order to make use of SMACH and such that the control logic code templating and code generation paradigm. It has been of the demo could be specified using a state machine. After demonstrated on a simulated humanoid robot platform in that it was possible to design the necessary code templates two different experiments. and script the demo using SMACHA. Once the custom tem- plates and the SMACHA script had been created for the first 5. ACKNOWLEDGEMENTS demo, it was possible to reuse both of them to very rapidly This work has been funded by the Horizon 2020 ICT-FoF script the second experiment for block stacking. In both Innovation Action no 680431, ReconCell (A Reconfigurable cases, it was possible to run the Python SMACH code gen- robot workCell for fast set-up of automated assembly pro- erated by SMACHA without further modification with both cesses in SMEs) and by the GOSTOP programme, contract experiments completing successfully. no C3330-16-529000, co-financed by Slovenia and the EU under the ERDF. The Baxter SMACHA package3 includes the following cus- tom code templates: BaxterBase (extends the core Base 6. REFERENCES template), LoadGazeboModelState (allows allows a specified [1] Jinja2 (The Python Template Engine). Gazebo model to be loaded into the simulator), MoveTo- http://jinja.pocoo.org/. (Last accessed 2017-07-24). JointPositionsState (moves a Baxter limb to a specified set [2] YAML Ain’t Markup Language (YAMLTM) of joint positions), PoseToJointTrajServiceState (uses in- Version 1.1. http://yaml.org/spec/1.1/. (Last verse kinematics to calculate a set of joint positions from accessed 2017-07-25). a specified end-point pose), and GripperInterfaceState (ei- [3] J. Bohren and S. Cousins. The SMACH High-Level ther opens or closes a specified gripper). In the inital states Executive [ROS News]. IEEE Robotics Automation of the pick and place experiment state machine, as specified Magazine, 17(4):18–20, Dec. 2010. by the SMACHA script in Listing 1, a table model must [4] E. Guizzo and E. Ackerman. How rethink robotics built be loaded into the simulator using the LoadGazeboModel- its new baxter robot worker. IEEE spectrum, page 18, State template, followed by a block model placed at a spec- 2012. ified pose on the table, and the left limb of the robot must be moved to a starting position using the MoveToJoint- [5] P. Schillinger, S. Kohlbrecher, and O. von Stryk. PositionsState template. Subsequently, the robot enters a Human-robot collaborative high-level control with “PICK BLOCK” state as specified by the “pick block” sub- application to rescue robotics. In 2016 IEEE International Conference on Robotics and Automation 2Video available at: https://youtu.be/WFp keDsA6M (ICRA), pages 2796–2802, May 2016. 3https://github.com/abr-ijs/baxter smacha 22 Extending the Workspace of Pseudo-Linear Variable-Lever Variable Stiffness Actuator Miha Dežman Andrej Gams Institute Jožef Stefan Institute Jožef Stefan Jamova cesta 39 Jamova cesta 39 Ljubljana, Slovenia Ljubljana, Slovenia miha.dezman@ijs.si andrej.gams@ijs.si ABSTRACT Another advantage is the simplicity of mechanical configuration. Among the different mechanically compliant actuators, variable However, a drawback of the device is the growing pressure angle stiffness actuators have the possibility to change their mechanical at lower stiffness positions, which increases the torque required to compliance on the fly, which is favorable in many force control change the stiffness and limits the devices maximum deflection. applications. Though their control and mechanics are of higher We begin the article with a description of the original operation complexity, their simplification is in the research focus of many principle in section 2 and explain the modification in section 3. In groups. In this work we propose a modification by adding addi- section 4 we explain the parameter optimization and show its re- tional springs to our novel variable stiffness principle, which solves sults in section 5. We conclude the work in section 6. the rising stiffness torque drawback of the previous design. We per- formed a parameter search over two modification parameters to find 2. VSA OPERATION PRINCIPLE the combination with the most suitable workspace. The proposed The development of a variable stiffness actuator (VSA) encom- mechanical modification extends the torque/deflection workspace passes a complex design process with many parameters [10]. The of the original principle, while keeping its favorable properties, number of parameters is higher than with classical stiff actuators. i.e. pseudo-linear torque deflection characteristics and low power Furthermore, mechanical convenience is also important to keep the to change stiffness. design simple and easy to manufacture, and is hard to achieve due to so many parameters. The devices requirements also differ based Keywords on the desired application. mechanically compliant actuator, variable stiffness, optimization, The goal of our design is to keep the structure simple, with a favor- parameter search able torque deflection characteristic. The weight of the device and its power requirements should also be minimal. 1. INTRODUCTION The core of our device is a combination of a cam mechanism and a variable lever principle (see Fig. 1). A curved lever arm rotates Robots using mechanical compliance have several advantages when (ϕ compared to classical stiff actuators [1], including safer human- d) around the pivot A and compresses the spring follower. This provides a reaction torque to the external load. Via the follower robot interaction and higher energy efficiency. The compliant ele- rotation (ϕs) around pivot B, the effective radius of the spring to ment is a low-pass filter [2] that reduces the peak gear forces, how- the pivot A changes, thus changing the stiffness of the mechanism. ever, at the same time it also reduces the bandwidth of the actuator. The system is in principle unidirectional, since the spring can be Improved force accuracy in higher stability is another advantage. compressed only. To achieve a bidirectional application, a system Much of the research is focused in this field because of the favor- of gears or cables can be used to change the bidirectional motion of able properties of such devices. Higher mechanical and control the external link to unidirectional motion for spring compression. complexity offer many simplification possibilities in combination At zero deflection, the follower can be rotated with minimal torque. with a wide array of possible applications. Researchers focus on The reason is, that the device uses a pretension-less spring princi- the novel device architectures development and prototype manu- ple [10]. facturing due to commercial unavailability of such devices. We The analytical model of the torque/deflection characteristics was point the reader to some articles that provide an overview over dif- ferent design architectures, for example, [3], [4] and [5]. There already exist many interesting applications that utilize the principle of mechanical compliance. One of them is a small jump- ing robot called Salto [6], that is capable of jumping to a 1m height, trying to mimic a jumping animal called Senegal bushbaby (Galago senegalensis). Another interesting example is the passive-ankle ex- oskeleton [7], which is able to reduce the users walking effort for 10% in a passive manner without using motors as power inputs by utilizing springs. We developed a similar exoskeleton, where we implemented a mechanism to close the clutch using a small mo- tor [8], which increased its operation reliability. This work is the development continuation of a novel variable stiff- ness actuator principle presented in [9]. A quasi-linear torque- Figure 1: The actuator working principle: a) equilibrium posi- deflection is the main advantage of the proposed configuration. tion, b) deflection under external load, and c) varying stiffness. 23 Original concept sti derived in [9]. To test the principle, a rapid-prototype was also de- ffness torque 40 veloped. In this article, we consider a case of a real prototype, with ]m ϕs = 5o the theoretical data specified in Table 1. [N ϕ 30 s = 10o e ϕs = 20o u ϕs = 30o rq 20 ϕ to s = 50o Table 1: Calculation and prototype parameters ϕs = 80o ess ϕs = 90o Variable Name Value Unit n 10 tiff Rs Cam curve radius 50 [mm] S k 0 lin Linear spring stiffness 72.6 [N/mm] 0 5 10 15 20 25 30 35 40 ϕd Deflection angle (min/max) ± 40 [o] Deflection angle (ϕd) [deg] ϕs Stiffness setup angle range 0 - 90 [o] Tpn Pos. motor nominal torque 15 [Nm] Figure 3: The torque required to change the device stiffness for Tpp Pos. motor peak torque 22.5 [Nm] the original case. Tsn Stiff. motor nominal torque 5 [Nm] Tsp Stiff. motor peak torque 6.3 [Nm] 3. EXTENDING THE VSA WORKSPACE In order to extend the torque/deflection workspace, we propose to Due to nominal and peak torque limits of the position motor and add two more springs into the system. With this we are able to keep stiffness motor gearboxes, a torque/deflection graph for the selected the torque required to change stiffness low while extending the de- mechanism with marked operation areas can be created. The blue vices workspace. The modification is shown in Fig. 4. area marks the workspace where both motors operate below the nominal torque, while the red area represents the torque levels be- tween the nominal and peak torque limits. Figure 4: The main device parameters (left) with the two pro- posed modifications (right). The first modification is the addition of a torsion spring ktor (see Fig. 4). The stiffness torque at zero deflection is negligible and it rises with the deflection, whereas it is higher at lover stiffness posi- tions and lower at high stiffness positions. We can thus increase the workspace by adding a torsion spring to create a negative stiffness torque offset. This way, the compensation stiffness torque (Mcs), Figure 2: Normal torque/deflection workspace and the corre- due to spring ktor, rises linearly from ϕs = 90o to ϕs = 5o as de- sponding stiffness graph. scribed in equation M One drawback of the proposed system is the rising pressure angle 0 Msc(ϕs) = · (90o − ϕs) , (1) 90o − 5o at lower stiffness presets and higher deflections. Due to the cur- vature of the deflection lever, no torque is theoretically needed to where the Mcs is the compensation torque, T0 is the maximum change the stiffness at zero deflection. However, as the deflection torque and ϕs the stiffness angle. The maximum stiffness compen- increases, the stiffness torque rises due to an increasing pressure sation torque is reached at ϕs = 5o. The workspace below ϕs = 5o angle. This is one of the drawbacks of the current system architec- is deemed unusable. By calculating the new stiffness torque, an in- ture. crease in the workspace can be observed as seen Fig. 5. The black The stiffness torque required to change the stiffness is shown in area represents the workspace before and the (red and blue area) Fig. 3 for different stiffness presets. One can see that the stiffness after the addition of the torsion spring. motor torque theoretically easily gets higher than 30 Nm, which Second modification is the addition of a linear spring (klin2), which is above the peak torques of the stiffness motor and the position is not as trivial as before. It is best, if the torque required to assist motor. Though a stronger motor could be used, a mechanical so- the stiffness motor comes directly from the external torque. There- lution would be far more convenient. Seeing Fig. 2, the torque fore, the second modification is a spring connected between the limits separate the torque/deflection graph into three areas. The curved deflection arm and the follower. Note, that since the fol- yellow workspace area is where the stiffness torque overpowers the lower can be rotated, the spring system is compression based only. stiffness motor. It is possible to stay outside of the yellow area by This way, at lower stiffness angles the assistance is the strongest, keeping the stiffness angle above ϕs > 40o. Here the actuator op- while at the higher stiffness presets the assistance is minimal or erates reliably. However, by implementing two additional springs, there is none. The combination of the L1 and klin2 parameter has the workspace can be extended. an optimal solution. To find it, we need a new mathematical model. 24 The gray (black) area represents the workspace, where the stiffness Figure 5: Normal deflection torque (black area) and the new workspace with torsion spring (red-blue area). The corre- sponding stiffness torque is on the right graph. Figure 7: Workspace comparison between the normal and modified mechanism version, where red-blue area is desired Considering the two additional springs, the new torque/deflection and black undesired. characteristics can be calculated as M setup motor does not have enough torque to move. One can see, dnew = Md + Mdk2, (2) that by addition of the springs, the black area shrinks. In the blue where the Md presents the torque from the old system, derived in area, the motor torque is below nominal and in the red, the mo- [9]. The torque needed to compress the new spring klin2 is Mdk2 tor torque is between nominal and peak operation limits. The size and can be calculated as a vector product of the areas can be numerically calculated and compared for dif- ferent parameter combinations. The parameters Tpn, Tpp, Tsn and M dk2 = ~r01 × ~ Fk2 , (3) Tsp represent the limits of the position motors and stiffness motors gearboxes. Their values can be found in Table 1. These parameters where the ~ Fk2 is the force from the klin2 spring compression and limit the torque/deflection workspace of our actuator. ~r01 the lever vector, as can be seen from Fig. 6. The new stiffness torque is calculated as Msnew = Ms − Msc − Msk2, (4) where the Ms is adapted from [9] and the Msc from (1). The Msk2 represents the stiffness torque resulting from the spring compres- sion and is calculated as the vector product M sk2 = ~r34 × ~ Fk2 . (5) Again, see Fig. 6. Note, that when the stiffness motor rotates (ϕs), the spring klin2 is only active when the distance between points (x4, y4) and (x1, y1) is smaller than the no-load spring length. 4. PARAMETER OPTIMIZATION In order to determine the optimal parameters r and klin, we per- formed a parameter search. Fig. 7 represents the workspace for normal PLVL-VSA configuration and the one with additional springs. Figure 8: Black, red-blue area sizes and the corresponding cost value for different klin and L1 parameter combinations. Our goal is to find the optimal combination of parameters L1 and klin2, where the black area of the actuator is of minimal size and the red-blue area is of maximal size. We cycle through both parame- ters in range L1 = 10 − 40mm and klin2 = 1 − 20N/mm. For each parameter set of L1 and klin2, we use the theoretical model and cal- culate the torque/deflection graph with nominal/peak torque limits. We numerically measure the areas of the workspace areas and col- lect them into a matrix. We determine the optimal configuration by taking the normalized versions of areas using the following cost function Figure 6: A scheme of the proposed mechanism with the force and distance vectors. δ = Aredblue ∗ (1 − Ablack)10, (6) 25 where Aredblue represents the area of the sum of red and blue work- 6. CONCLUSIONS spaces and the Ablack represents the black workspace. The 10th We presented the continuing development of a novel variable stiff- exponential makes the local maxima easier to see. The optimiza- ness actuator. The proposed modifications of the original operation tion graphs are shown in Fig. 8. Looking at the graph titled Cost principle extends the devices torque/deflection workspace, while value, one can spot a local maxima near parameter combination keeping the same stiffness and position motors. The modification L1 = 25 mm and klin = 5.5 N/mm. A rounded approximation close successfully minimizes the effects of the original principles main to the real optimal value is better suited for manufacturing. At the drawback, the rising stiffness torque due to the rising pressure an- chosen parameters, the sum of red-blue area is at maximum while gle. In the future, the proposed modifications will be implemented the black area is at minimum. in a real-world prototype. 5. OPTIMIZED WORKSPACE 7. REFERENCES The optimal parameter torque/deflection graphs are shown in Fig. 9. [1] Ronald Van Ham, Thomas G Sugar, Bram Vanderborght, One can see, that the system still preserves the quasi-linearity, which Kevin W Hollander, and Dirk Lefeber. Compliant actuator designs. IEEE Robotics & Automation Magazine, 16(3), 2009. [2] Gill A Pratt and Matthew M Williamson. Series elastic actuators. In Intelligent Robots and Systems 95.’Human Robot Interaction and Cooperative Robots’, Proceedings. 1995 IEEE/RSJ International Conference on, volume 1, pages 399–406. IEEE, 1995. [3] Bram Vanderborght, Alin Albu-Schäffer, et al. Variable impedance actuators: A review. Robotics and autonomous systems, 61(12):1601–1614, 2013. [4] Nevio Luigi Tagliamonte, Fabrizio Sergi, et al. Double actuation architectures for rendering variable impedance in compliant robots: A review. Mechatronics, 22(8):1187–1203, 2012. [5] Ronald Van Ham, T Sugar, et al. Review of actuators with passive adjustable compliance/controllable stiffness for robotic apllications. IEEE Robotics and Automation mag., Figure 9: The torque/deflection workspace for the new mecha- 16(3):81–94, 2009. nism with the corresponding stiffness. [6] Duncan W Haldane, MM Plecnik, Justin K Yim, and Ronald S Fearing. Robotic vertical jumping agility via is favorable from the control perspective. The deflection range is series-elastic power modulation. Science Robotics, now extended to 40o at lower stiffness presets. As can be observed, 1(1):eaag2048, 2016. we can vary the stiffness between 30 to 200 Nm/rad. The stiffness [7] Steven H Collins et al. Reducing the energy cost of human torque, required to change the stiffness, is shown in Fig. 10. As walking using an unpowered exoskeleton. Nature, predicted, due to the modifications, the new stiffness torque is kept 522(7555):212–215, 2015. between the peak torque limits. This shows that the proposed mod- [8] Miha Dežman, Jan Babič, and Andrej Gams. Qualitative ifications increases the performance of the device while using the assessment of a clutch-actuated ankle exoskeleton. In same motors as in the original concept. Since the motors contribute International Conference on Robotics in Alpe-Adria Danube a lot of weight to the overall system, keeping the motors small also Region, pages 778–786. Springer, 2017. helps keeping the overall device light, small and compact. [9] Miha Dežman and Andrej Gams. Pseudo-linear variable lever variable stiffness actuator: Design and evaluation. In New modified stiffness torque Advanced Intelligent Mechatronics (AIM), 2017 IEEE International Conference on 6 , pages 785–790. IEEE, 2017. ] [10] S. Wolf, G. Grioli, O. Eiberger, et al. Variable stiffness 4 m actuators: Review on design and components. IEEE/ASME [Ne 2 Transactions on Mechatronics, 21(5):2418–2430, Oct 2016. urq 0 to ess -2 n tiffS -4 -6 0 5 10 15 20 25 30 35 40 Deflection angle (ϕd) [deg] ϕs = 5o ϕs = 10o ϕs = 20o ϕs = 50o ϕs = 80o ϕs = 90o Figure 10: The stiffness torque of the modified mechanism. 26 Soft Humanoid Robots Koh Hosada Department of System Innovation, Graduate School of Engineering Science, Osaka University 1-3 Machikaneyama, Toyonaka, Osaka 560-8531/ Japan hosoda@sys.es.osaka-u.ac.jp ABSTRACT A human can exhibit intelligent behavior. Yet, we do not completely understand the mechanism how the behavior emerges. Adaptive behavior is obviously generated by the brain, but brain alone cannot explain everything. Key components are soft tissue, muscles, bones, and skin. This talk will introduce our challenges to build soft humanoid robots consisting of muscles, bones, and skin so that we can constructively understand the human’s adaptive intelligence. 27 Semantic Reasoning under Realistic Conditions Karinne Ramirez-Amaro and Gordon Cheng Institute for Cognitive Systems, Technical University of Munich, Germany Arcisstrasse 21, 80333, Munchen karinne.ramirez@tum.de and gordon@tum.de ABSTRACT One fundamental issue of autonomous robots in task do- mains is its capability to learn new skills and to re-use past experiences under different situations as efficient, intuitive and reliable as possible. A promising mechanism to achieve that is via learning from demonstrations or observations. In this abstract, we present a novel learning method that gen- erates compact and general semantic models to infer human activities. We propose a method that allows robots to obtain and determine a higher-level understanding of a demonstra- tor’s behavior via semantic representations [4]. First, the low-level information is extracted from the sensory data, then a meaningful semantic description, the high-level, is obtained by reasoning about the intended human behaviors [6]. The introduced method has been assessed on different robots, e.g. the iCub [3], REEM-C [5], and TOMM [2], with different kinematic chains and dynamics. Furthermore, the Figure 1: Main modules of our framework for the robots use different perceptual modalities, under different segmentation and recognition of human everyday ac- constraints and in several scenarios ranging from making tivities using multiple sensors. a sandwich to driving a car assessed on different domains (home-service and industrial scenarios). Each of the studied scenarios poses distinct and challenging levels of complexity Reality Performances. IEEE/RSJ International to demonstrate, that our method does not depend on the Conference on Intelligent Robots and Systems (IROS). analyzed task, thus presenting a major benefit compared to IEEE, Sept 2017. classical reasoner approaches. Another important aspect of [2] E. Dean, K. Ramirez-Amaro, F. Bergner, I. Dianov, our approach is its scalability and adaptability toward new P. Lanillos, and G. Cheng. Robotic technologies for fast activities, which can be learned on-demand. deployment of industrial robot systems. In 42nd IEEE Industrial Electronics Conference (IECON). IEEE, Our semantic reasoning method can be extended to a more October 2016. higher level for the generation of tasks. Since our system is [3] K. Ramirez-Amaro, M. Beetz, and G. Cheng. able to on-line recognize and learn new activities on-demand, Understanding the intention of human activities thus we can produce a graph of all related activities that pro- through semantic perception: observation, duce a task. Therefore, our learning system extracts general understanding and execution on a humanoid robot. task structures which together with the obtained knowledge Advanced Robotics, 29(5):345–362, 2015. can improve and accelerate the teaching of new tasks. Fur- [4] K. Ramirez-Amaro, M. Beetz, and G. Cheng. thermore, we improved and validate this approach using a Transferring skills to humanoid robots by extracting Virtual Reality system, which presents a realistic, cluttered, semantic representations from observations of human space in which a variety of tasks can be accomplished. Us- activities. Artificial Intelligence, 247:95–118, June 2017. ing task graphs also allowed the robot to utilize sparse sets [5] K. Ramirez-Amaro, E. C. Dean-Leon, and G. Cheng. of instructions to construct the complete set of steps neces- Robust semantic representations for inferring human sary to carry out complex tasks [1]. Overall, the presented co-manipulation activities even with different compact and flexible solutions are suitable to tackle complex demonstration styles. In Humanoids, pages 1141–1146. and challenging problems for autonomous robots. IEEE, 2015. [6] K. Ramirez-Amaro, H. N. Minhas, M. Zehetleitner, M. Beetz, and G. Cheng. Added Value of 1. REFERENCES Gaze-Exploiting Semantic Representation to Allow [1] T. Bates, K. Ramirez-Amaro, T. Inamura, and Robots Inferring Human Behaviors. ACM Transactions G. Cheng. On-Line Simultaneous Learning and on Interactive Intelligent Systems, 7(1):5:1–5:30, 2017. Recognition of Everyday Activities from Virtual 28 Integrating Multi-modal Tactile Signals to a Compliant Control to Improve physical HRI Emmanuel Dean-Leon and Gordon Cheng Institute for Cognitive Systems, Technical University of Munich, Germany Arcisstrasse 21, 80333, Munchen dean@tum.de and gordon@tum.de ABSTRACT a) Multimodal b) Self-organizing Motor Babbling c) 3D Tactile Signals Network Reconstruction LED ro Proxi xim mity it L y ED Norma Force Skin Patch The development of breakthrough technologies helps the de- PC ID4 ID2 ID5 Root ployment of robotic systems in the industry. The imple- Inertia cceleration Temperature ID3 ID1 Cell Temp UART 4 Mbit/s tch Root Cell Tactile mentation and integration of such technologies will improve Ethernet Section Swi 1 Gbit/s Unit productivity, flexibility and competitiveness, in diverse in- d) Patch e) DH-like f) Robot Parametric g) Skin Torque Localization Parameters Dynamic Model dustrial settings specially for small and medium enterprises. R In this talk, we present a framework [1] that integrates three C oot el Skin l Control novel technologies, namely safe robot arms with multi-modal Torque Resolver and auto-calibrated robot skin [4], an end-to-end approach g) SkinTorque Control Fusion for transforming multi-modal tactile signals into a compli- ant control to generate different dynamic robot behaviors Torque Arm Resolver Control Tactile [2], and an intuitive and fast teaching by demonstration (e.g. Interaction Cartesian, Joint,etc.) Arm Low method that segments and recognizes the robot activities Level Interface on-line based on re-usable semantic descriptions [6]. A key UR Control Unit Standard Robot Control Interface component of our framework is a robot parametric modeling Torque Resolver (Torque to Position/Velocity) Nonlinear Trajectory Generator Inner Velocity Control based on the artificial skin multi-modal sensors (proximity, force and acceleration) [5]. We validate our approach in our Control Fusion Command robot TOMM [3] with an industrial application. Interface Position Interface Velocity Interface Torque Interface Internal Interface Figure 1: Main modules of our approach. 1. REFERENCES [1] E. Dean, K. Ramirez-Amaro, F. Bergner, I. Dianov, P. Lanillos, and G. Cheng. Robotic technologies for fast deployment of industrial robot systems. In 42nd IEEE IECON. IEEE, Oct. 2016. [2] E. Dean-Leon, F. Bergner, K. Ramirez-Amaro, and G. Cheng. From multi-modal tactile signals to a compliant control. In 2016 IEEE-RAS Humanoids. [3] E. Dean-Leon, B. Pierce, F. Bergner, P. Mittendorfer, K. Ramirez-Amaro, W. Burger, and G. Cheng. Tomm: Tactile omnidirectional mobile manipulator. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 2441–2447, May 2017. [4] P. Mittendorfer, E. Dean, and G. Cheng. 3D spatial self-organization of a modular artificial skin. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3969–3974, Sept 2014. [5] P. Mittendorfer, E. Dean, and G. Cheng. Automatic robot kinematic modeling with a modular artificial skin. In 2014 IEEE-RAS International Conference on Humanoid Robots, pages 749–754, Nov 2014. [6] K. Ramirez-Amaro, M. Beetz, and G. Cheng. Transferring skills to humanoid robots by extracting semantic representations from observations of human activities. AI, 247:95–118, June 2017. 29 Event-Driven Systems for Efficient Reactive Control with Large Scale Robot Skin Florian Bergner, Emmanuel Dean-Leon and Gordon Cheng Institute for Cognitive Systems, Technical University of Munich, Germany Arcisstrasse 21, 80333, Munchen florian.bergner@tum.de, dean@tum.de and gordon@tum.de ABSTRACT Tactile human-robot interaction is essential for collaborative robots in industrial, health-care and household application scenarios and increases the robot’s intuitiveness, flexibility and safety. One key element of enabling intuitive interac- tions is the ability to manually guide the robot by touching it for e.g. in teach-in scenarios [4]. A promising approach to upgrade existing robot systems with such abilities is to use skin for robots and implement multi-contact controllers. For taking full advantage of skin, the skin has to cover the robot completely. However, the application of large scale skin induces new challenges: 1) solving issues in transmit- ting huge amounts of tactile information with low-latency and 2) processing huge amounts of tactile information in Figure 1: The robot TOMM [5] with two arms and real-time. In this abstract we present our new event-driven two grippers covered with skin; the robot holds a approach to tackle these challenges. We developed a novel paper towel in its left gripper which it uses to push multi-modal event-driven robot skin [1, 2] and combined it the right arm; the right arm is controlled by our with our novel efficient event-driven reactive skin controller novel event-driven tactile reaction controller; it tries for large scale robot skin [3]. Event-driven systems only to avoid contacts and moves to the right. sample, transmit and process information when the novelty of the information is guaranteed. This increases their effi- 1. REFERENCES ciency in comparison to synchronous systems. We evaluated [1] F. Bergner, P. Mittendorfer, E. Dean-Leon and G. our system in a comprehensive performance evaluation with Cheng, “Event-based signaling for reducing required our robot TOMM. TOMM has two UR5 robot arms, each data rates and processing power in a large-scale covered with 253 multi-modal skin cells. The results show artificial robotic skin”, in IEEE/RSJ International that the event-driven reactive skin controller always outper- Conference on Intelligent Robots and Systems (IROS), forms the synchronous reference controller while both con- pp. 2124-2129, 2015. trollers show exactly the same response. When the robot [2] F. Bergner, E. Dean-Leon and G. Cheng, is not moving then the event-driven controller reduces the “Event-based signaling for large-scale artificial robotic CPU usage by 78% in comparison to the synchronous refer- skin - Realization and performance evaluation”, in ence controller. When the robot is responding to contacts IEEE/RSJ International Conference on Intelligent then the CPU usage reduces by 66%. Robots and Systems (IROS), pp. 4918-4924, 2016. [3] F. Bergner, E. Dean-Leon and G. Cheng, “Efficient Event-Driven Reactive Control for Large Scale Robot Skin”, in IEEE/RSJ International Conference on Robotics and Automation (ICRA), pp. 394-400, 2017. [4] E. Dean-Leon, K. Ramirez-Amaro, F. Bergner, I. Dianov, P. Lanillos and G. Cheng, “Robotic Technologies for Fast Deployment of Industrial Robot Systems”, in The 42nd Annual Conference of IEEE Industrial Electronics Society (IECON), pp. 6900-6907, 2016. [5] E. Dean-Leon, B. Pierce, P. Mittendorfer, F. Bergner, K. Ramirez-Amaro, W. Burger and G. Cheng, “TOMM: Tactile Omnidirectional Mobile Manipulator”, in IEEE International Conference on Robotics and Automation (ICRA), 2017, In Press. 30 Indeks avtorjev / Author index Bem Martin .................................................................................................................................................................................. 15 Berger Florian .............................................................................................................................................................................. 30 Bevec Robert ................................................................................................................................................................................ 15 Cheng Gordon .................................................................................................................................................................. 28, 29, 30 Dean-Leon Emmanuel ............................................................................................................................................................ 29, 30 Deniša Miha ................................................................................................................................................................................. 11 Dežman Miha ............................................................................................................................................................................... 23 Gams Andrej ............................................................................................................................................................................ 7, 23 Gašpar Timotej ............................................................................................................................................................................. 15 Gosar Žiga .................................................................................................................................................................................... 15 Hosada Koh .................................................................................................................................................................................. 27 Kovač Igor .................................................................................................................................................................................... 15 Ramirez-Amaro Karinne .............................................................................................................................................................. 28 Ridge Barry ............................................................................................................................................................................ 15, 19 Ude Aleš ............................................................................................................................................................................. 7, 11, 15 31 32 Konferenca / Conference Uredila / Edited by Robotika / Robotics Andrej Gams, Aleš Ude Document Outline A - Naslovnica-SPREDNJA - H B - Naslovnica - notranja - H C- Kolofon - H D-E - IS2017 - skupni zacetni del Blank Page F - Kazalo - H G - Naslovnica podkonferenca - H H - Predgovor - H I - Programski odbor - H J - PDF - H 01_Robotics_IS 02_Robotics_IS 03_Robotics_IS 04_Robotics_IS 05_Robotics_IS 06_Robotics_IS 07_Robotics_IS 08_Robotics_IS 09_Robotics_IS K - Index - H L - Naslovnica-ZADNJA - H Blank Page Blank Page Blank Page Blank Page Blank Page Blank Page