TMCnet News
Combination of Robot Simulation with Real-time Monitoring and Control [Sensors & Transducers (Canada)](Sensors & Transducers (Canada) Via Acquire Media NewsEdge) Abstract: The paper mainly focuses in combining virtual reality based operation simulation with remote real-time monitoring and control method for an experimental robot. A system composition framework was designed and relative arm-wheel experimental robot platform was also built. Virtual robots and two virtual environments were developed. To locate the virtual robot within numerical environments, relative mathematical methods is also discussed, including analytic locating methods for linear motion and self-rotation, as well as linear transformation method with homogeneous matrices for turning motion, in order to decrease division calculations. Several experiments were carried out, trajectory errors were found because of relative slides between the wheel and the floor, during the locating experiments. Writing-monitoring experiments were also performed by programming the robotic arm to write a Chinese character, and the virtual robot in monitoring terminal perfectly followed all the movements. All the experiment results confirmed that virtual environment can not only be used as a good supplement to the traditional video monitoring method, but also offer better control experience during the operation. Copyright © 2014 IFSA Publishing, S. L. Keywords: Numerical robot, Real-time monitoring, Remote operation, Virtual reality. (ProQuest: ... denotes formulae omitted.) 1. Introduction In remote operations, it is a significant functional parameter that how much the end effecter would follow the human instruction sent from the controller. Traditional remote control mechanism with video feedback can offer lower lags that simplify the remote operations, usually thanks to its dedicated data transferring cables. But, if there were a large increment of physical distance between the end effecter and operator, or the end effecter were working in extreme environments, the traditional video feedback strategy would probably encounter problems as: 1) hard to acquire video information; 2) hard to ensure control accuracy; 3) hard to control end effecter; 4) high cost for dedicated data lines. Many endeavors for solving relative problems have been done by researchers all over the world: J. Aleotti, S. Caselli and M. Reggiani from Italy developed a method for teaching industrial robot by recognizing human natural movements based on VR and computer vision technology [1], but they did not go forward to remote control of the robot; Jee-Hwan Ryu, Jordi Artigas and Carsten Preusche who have improved remote control performance by developing a bi-directional passive control strategy for varying time delay remote control applications [2]; Chiristain Smith et al. [3], Aneesh N. Chand et al. [4], and Alberto José Álvares et al. [5] have also studied on remote control of industrial robots or machine tools separately using Ethernet or VR technology. In China, Xuejian He and Yonghua Chen from the University of Hong Kong successfully improved industrial robot off line programming and path planning problem with haptic devices and VR [6], but they didn't work on remote control ether; Jian Zainan et al from Harbin Institute of Technology realized remote control of a space robot for satellite service with adaptive control and robustness against model errors [7]; and in 2003, Gu Minqiang et al., from Shanghai Jiao Tong University cooperated with Maro Ravera et al., from Academy of Robotics, Politécnico di Milano, Italy in remote controlling a robot using VR through internet, but their article did not mention whether or how their system feedback the robot working status or how the data transferring delay affected the application [8]. In this article we followed the step of former researches above and forward trying to combine robot pre-work simulation with real-time monitoring and control, in order to develop a less expense but more accurate and easy way for robot remote control. A VR based remote control and monitoring system has been built; Several experiment on robot locating, real time monitoring and control have been done, and the result showed that VR technology can not only act as an effective add on to video monitoring methods, but also bring much more better controlling experience in remote operation, meanwhile it offers better assistant effect with the same data transferring bandwidth compared to video feedback strategies. 2. Experimental System Composition and Relative Mathematics 2.1. System Composition Our VR based simulation and monitoring system totally has four components, as in Fig. 1, mainly including system control terminal, monitoring terminal, and an experimental robot. System control terminal is the command center of the whole system; it's composed by one or two computer and a control panel. The operator uses control panel to generate control commands, and these commands are consequently sent to the computer which virtual 5-DOF robot run on. Virtual robot has the same mechanical structure with the experimental robot, and is directly driven by control commands. It can ether run solo as a off line simulator of the robot or run online as a predictor of the real robot, by comparing working status to the other virtual robot on monitoring terminal. 2.2. Robot Locating Method in Virtual Environment There are two parts of kinematics for the experimental robot, one is for differential mobile platform, and another one is for robotic arm. At the very beginning of every step of computing, the initial position is supposed to be a know parameter, and then a calculating coordinate could be set up relative to the initial position. In Fig. 2, (Xo, Yo, O) is global coordinate, (X, Y, P) is the local coordinate of the mobile platform, Px is the positive forward direction of the robot, and we also suppose Px and X are in the same direction when t=ti. Consequently, when t=U+l, new position and orientation can be described by X, Y, and 0, meanwhile, these three parameters can also be used for locating the virtual robot in virtual environment. As mentioned, we treat robot position and posture (Xt, Yt, 0t) on t=ti as known parameters, and consequently, new position and posture parameters (Xt+i, Yt+i, 0t+i) on t=ti+1 can be calculated with wheel speeds vi and vr, of the moving platform, and the moving time At=ti+i-tj. For simplifying the calculation, we classified the robot motion into three types, including direct line motion, when the two wheels have the same speed and direction (vi=vr); self rotation when the two wheels have the same speed but inverse direction (vi=-vr); and turning motion when the two wheel speeds are not equal but always keeping a constant difference (v/-vr=C). Then, for direct line motion, the new position and posture of the robot on t=U+i can be calculated as equation (1). ...(1) For self rotation, the new position and posture of the robot on t=ti+i can be calculated as equation (2). ...(2) For turning motion, we introduced homogeneous matrices for simplifying the computing process, to decrease dividing calculation quantity accompanying with the analytic method above and accelerate the simulation speed. As shown in equation (3), we built a homogeneous matrix Ot using robot position and posture parameters in global coordinate (Xt, Yt, 6,) on t=U first, and then the motion quantities in local coordinate of robot in the following time period At can be described as equation (4), in which Ax, Ay and AO can be replaced with expressions in equation (5). ...(3) ...(4) ...(5) Consequently, the new position and orientation matrix Ot+i in global coordinate can be acquired by calculating equation (6). ...(6) Actually, we only need to calculate the first two items in third column of matrix Ot+i, and A6+0t as well. Finally, new (Xt+i, Yt+i, 0t+i) quantities can be acquired, and we then send them to the virtual robot running on monitoring end. 2.3. Robotic Arm Animating Method As shown in Fig. 3, the robotic arm is composed by base part, arm part 1, arm part 2, arm part 3 and a griper part, with every joint may rotate in a 180degree-range. Accordingly, six local coordinates are created exactly at the rotation center of every joints, a global coordinate is also set up and placed coincide with the local coordinate of the base part, as in Fig. 4. In animating the virtual robot of system control end, rotating angles of six local coordinates are exactly translated from rotating angles of six potentiometers on the control panel. With these angles, forward kinematics is applied for following calculation. According to D-H method, homogeneous translation from coordinate i-1 to i can be operated as equation (7). In equation (7), ai, di, ai and 0i are separately representing part length (between two joints), joint offset, part twisting angle, and joint angle. In which, i is part number, C and S are separately abbreviations of trigonometric functions cosine and sine. Finally the homogeneous translation from base part to the griper is as equation (8). ...(7) ...(8) The reason we directly mapping rotation angles from potentiometers on control panel to virtual robots was that there no mass would affect the movement of the virtual robots in VR environment. Every movement acted highly constant with the commands or feedback data received, which made directly mapping the most efficient and fastest way for realtime visualization. 3. System Implementation 3.1. Experimental Robot Modeling We modeled two versions of experimental robot, one is simplified model, and another one is regular. The former one was used for real-time visualization on system control terminal, when the latter one was for better showing on monitoring terminal, in a Virtools created environment. It is very necessary to mention that these models might not be used directly in computer graphics (CG) libraries such as OpenGL Performer. It has to be firstly transformed into mesh or polygon formats like open flight (*.flt), Alias/Wavefront (*.obj) and etc., and then modified by correcting local coordinate origins of every moving components to their geometric center. This is because constrains established during modeling work in Solidworks has been removed in model transformation, but at same time CG programming would care very much on those origins. 3.2. Virtual Environments Establishment As mentioned above, our experimental system has two virtual environments. As shown in Fig. 5, the virtual environment of system control terminal was created by SGI OpenGL Performer, and Visual C++ Express for fast real-time visualization. The window on the left of Fig. 5 is command displaying interface, in which data lines automatically refreshed approximately 60 times in one second. Information includes commanding wheel speed and robotic arm joints angles. Window on the right is virtual robot, shows real-time postures according to the commands. This virtual robot can be used as a robot offline simulator or online movement predictor. As shown in Fig. 6, monitoring terminal virtual environment has been created by a trail version of Virtools, from Dassault Systems, for a better showing at the time. We built one our own dynamic linking library and loaded it for data communication, I/O and interactive functions; we also set four cameras in the scene for different viewpoints to avoid observing dead space. Data for animating virtual robot was accumulated by mentioned Arduino board on experimental robot, and transferred through wireless communication, under RS-232 protocol. Wireless hardwires and working frequencies were totally different between commanding and feedback channel, which insured no data conflict, and kept the system functioning. 3.3. System Control Terminal Setup The setup of system control terminal is shown in Fig. 7, it was configured to run on two computers. Parts of controlling end programs running on the left computer, with the screen displaying three GUIs separately for command sending, control panel data acquisition, and communication module driver programs. The Dell D620 laptop on the right is running virtual robot program, the data for real time animation was from serial connection between two computers. Controlling commands were transferred in broadcast mode, which means system control end could run solo with out real robot responding, so the virtual robot here could consequently run as a prework simulator. The experimental robot was actually the combination of one five degrees of freedom servo driven robotic arm and a differential moving platform. Each joint of the robotic arm can rotate to or stop at any angle with the resolution of 0.2 degree, in 180 degrees angular range. DC motors and rotary encoders are used for driving the whole platform and wheel speed sensing. As shows in Fig. 8, monitoring terminal is an independent computer on which a virtual robot runs for robot monitoring. The computer receives data of robot working status, and generates a real-time animation. Meanwhile, it also visualizes all the possible information the robot transferred out. This visualization will help keeping the robot and all the status of relative working places under the control of a human operator. 4. Experiments and Results We performed two series of experiments, one was for testing locating method for moving platform, and another one was for controlling and monitoring the robotic arm. Fig. 9 presents the moving platform locating experiment. During this experiment, we saw synchronization between virtual and real robots, but there was also trajectory error. This error was caused by relative slide between robot wheels and the floor, smaller when the robot was moving straight, whereas larger when turning. As experiment last longer, this sliding error would become more significant, especially when the robot was truing. Synchronization became much better when the robot was doing a slow self rotation; but as the speed increased; the error quantity also went larger. We tried to change the mounting position of the battery which was the heaviest part, and found the offset between robot mass center and rotating center would affect more significant than the wheel-floor slide at the same speed level when the robot was self rotating. Fig. 10 and Fig. 11 present two experiments for testing robotic arm synchronization performance and we received much better results compared with that of locating experiments. We performed a real-time writing experiment as in Fig. 10. In this experiment, we coded a script in order to generate control commands for robot writing. The commands was sent form system control terminal, then actuated by the robotic arm, the Arduino board was also successfully sensing the robot arm working status, and feeding back the data. In monitoring terminal, as we had expected, the virtual robot acted exactly as the real arm did, especially when the real one was stuck, and the virtual one also stopped and posed still the same with the real one. By comparing with two virtual robots of system control end and monitoring end, we could clearly to know where the arm was and where the arm was going to be, without directly looking at the real robot. As in Fig. 11, a shift moving experiment was also performed that the virtual robot was programmed to act sequential poses, moving green boxes from one place to the next; meanwhile, corresponding control commands were sending to the physical robot, and the former performed poses accordingly. At the end of experiment, we also tried to remote control the robot with simulated data lag, by inserting a random time delay function in our command sending and data feeding back functions. Thus, virtual robot on system control end acted the fastest, the one on monitoring end naturally acted slowest, and the real robot acted between the former two. During the operation, virtual robot on control end was functioning as a predictor, and this predicting effect helped us overcoming the data lag, and brought us a much better experience than use only virtual robot on monitoring end. 5. Conclusions and Future Work During the past work, we tried to combine operating simulation with real time monitoring and control together. We built an experimental robot and two virtual robots separately for work simulation and real time monitoring as well. Together with the contribution of Arduino micro-controllers and wireless communication modules, an experimental system was constructed, several experiments were also carried out, and methods for locating and controlling the experimental robot have been discussed as well. Research and experiments' results confirmed that: 1) Virtual reality technology can be used in realtime controlling and monitoring, it can be an effective add on to video monitoring methods. 2) The predicting effect functioned by virtual reality can help to overcome data lag effect, and bring much batter experience in remote operation. 3) Monitoring with virtual reality can offer better assistant effect with the same data transferring bandwidth compared to video feedback strategies. In future work, we intend to apply this principal to an industrial robot, add inverse kinematics to control end, develop a natural interaction method with our novel haptic device, combine code simulation, human assisted path planning, as well as real-time control and monitoring functions. Acknowledgements The project was supported by the National Nature Science Foundation of China (Grant No. 51105070), Doctoral Start-up Foundation of Liaoning province (Grant No. 201120006), and Normal Project of Scientific and Technical Research Foundation of Department of Education of Liaoning Province (Grant No. L2013118). References [1] . J. Aleotti, S. Caselli, M. Reggiani, Leveraging on a virtual environment for robot programming by demonstration, Robotics and Autonomous Systems, Vol. 47, Issue 2-3,2004, pp. 153-161. [2] . Jee-Hwan Ry, Jordi Artigas, Carsten Preusche, A passive bilateral control scheme for a teleoperator with time-varying communication delay, Mechatronics, Vol. 20, No. 7,2010, pp. 812-823. [3] . Christian Smith, Patrie Jensfelt, A predictor for operator input for time-delayed teleoperation, Mechatronics, Vol. 20, No. 7,2010, pp. 778-786. [4] . Aneesh N. Chand, Web-based tele-operated control system of a robotic vehicle, novel algorithms and techniques in telecommunications, Editors: Tarek Sobh, Khaled Elleithy, Ausif Mahmood, Mohammad A. Karim, Novel Algorithms and Techniques in Telecommunications, Automation and Industrial Electronics, 2010, pp. 32-36. [5] . A. J. Álvares, J. C. E. Ferreira, WebTuming: Teleoperation of a CNC Turning Center through the Internet, Journal of Materials Processing Technology, Vol. 179, No. 1-3,2010, pp. 251-259. [6] . Xuejian He, Yonghua Chen, Haptic-aided robot path planning based on virtual tele-operation, Robotics and Computer-Integrated Manufacturing, Vol. 25, 2010, pp. 792-803. [7] . Jiang Zainan, Liu Hong, Wang Jie, Huang Jianbin, Virtual reality-based teleoperation with robustness against modeling errors, Chinese Journal of Aeronautics, Vol. 22,2009, pp. 325-333. [8] . Gu Minqiang, Fan Xiumin, Ma Dengzhe, Maro Ravera, Alberto Rovetta, Telerobotic control through virtual environment, Machine Design and Research, Vol. 9, No. 3,2003, pp. 20-22. [9] . Tan Min, Xu De, Hou Zengguang, Wang Shuo, Cao Zhiqiang, Advanced robotic control, Higher Education Press, Beijing, 2007. Jianyu YANG, Hualong XIE, Xiufeng HAN, Wanshan WANG School of Mechanical Engineering and Automation, Northeastern University, Shenyang, 110819, P. R. China Tel: + 86-24-83687626 E-mail: [email protected] Received: 1 January 2014 /Accepted: 31 July 2014 /Published: 31 August 2014 (c) 2014 IFSA Publishing, S.L. |