TMCnet News

Study on Target Detection & Recognition Using Laser 3D Vision Systems for Automatic Ship Loader [Sensors & Transducers (Canada)]
[April 22, 2014]

Study on Target Detection & Recognition Using Laser 3D Vision Systems for Automatic Ship Loader [Sensors & Transducers (Canada)]


(Sensors & Transducers (Canada) Via Acquire Media NewsEdge) Abstract: This paper purposes a solution of the target detection and identification for automatic ship loader. For automatic ship loaders, the operation target should be detected and identified continuously and real-timely. By using the laser measurement systems (LMS), the ship cargo holds and the bulk cargo can be rebuilt as a group of 3D points. Then the image processing algorithm can identify the positions, sizes and shapes of the cargo holds and the bulk cargo from the 3D points. Based on the target information identified by the image processing algorithm, the ship loader can finish the loading operation automatically. At last, this paper describes and analyzes the experiment of the cargo height detection using LMS in Coal Terminal of Tianjin Port.



Copyright © 2013 IFSA.

Keywords: Laser measurement system, Image processing, Ship loader.


(ProQuest: ... denotes formulae omitted.) 1. Introduction With the accelerated process of economic development and globalization, the requirement of bulk cargo transportation is increasing. The seaborne volume of bulk cargo continues to grow faster and faster. The operation efficiency and the operation reliability of the bulk-cargo at the port become a topic of concern. Various technologies which can improve the operation efficiency and the operation reliability are put forward and applied. The automated operation of the bulk-cargo gradually becomes a trend [1-3].

At present, some large hub ports in the world have the specific practices and research of the bulkcargo automated handing and management. Many researches on increasing the degree of the automation of the bulk port focus on optimizing mechanicalresource allocation, intelligentialization of the management decision-making, technologies of port and other aspects. The number of the studies which are about the automation of the mechanical operation process is still relatively small [4].

Seagoing vessel's deadweight tonnage is large, and there are many different kinds of ship types which are not standardized. It is not easy to implement the automatic loading and unloading technology. Domestic and foreign researches on automatic loading and unloading of seagoing vessel are rare [5]. Therefore, to implement automatic loading and unloading of ships is a very forwarding-looking development direction. In this paper, the study object of loading and unloading is mainly seagoing vessel. The system knows the size of cargo hold and the shape of the bulk cargo hold according to the ship loader operating conditions. Then it can determine the relative position of ship loader and the hold and know when this loading point should be end. It can automatically control the loading position of the barrel, retention time and trajectory to achieve safe, efficient and balanced load.

Target detection and identification techniques of automatic ship loader include two key factors: the recognition of the hold size, such as the hold size and hold inclination, to determine hold loading flow and loading process; identification of the bulk cargo shape, such as the distribution and the height of the cargo, to determine the loading trajectory.

2. System Architecture and Module Design The automatic loading system combines laser 3D vision detection, image recognition, motion control and other advanced technology to implement the automatic bulk ship loader. Multi-axis servo motion system and LMS are installed on the ship loader's install base to implement the real-time scanning for the target. System structure as shown in Fig. 1. The LMS can rotate to different pose by using a servo motion system and the LMS can scan the 3D surface of the target in real time. By image processing algorithm running in the embedded controller, we can complete the recognition of the location, size, and inclination of the cargo hold and the shape of the bulk cargo. Via communication bus, the recognition data is sent to the master programmable logic controller to control the loading position of the barrel, retention time and trajectory to implement the automated operation.

3. TARGET Detection and Identification Algorithm 3.1. Cargo Hold Size Automatic Recognition Algorithm 1) The 3D object detection.

The application of target automatic detection technology is the prerequisite to achieve automatic running of the ship loader to perform loading and unloading operation. According to the port's environment, the LMS is selected to scan the surfaces of the target.

The working principle of LMS to measure distance is that the LMS emits a periodic frequency laser to a measured target and opens the pulse counter at the same time. The sinusoidal modulated laser irradiates the target and after being reflected, it was received by the receiver. The receiver receives the signal and closes the pulse counter at the same time. Finally, the signal processor compares the signal of the transmitting and receiving ends to obtain the phase difference a(f) [6, 7].

Assuming the time difference between transmitting and receiving signal is At , the modulation frequency is f, then we can obtain the round-trip time between the observation point and the measured point: (1) The distance between the observation point and the measured point is D which can be represented as: ... (2) where c is the speed of light, n is the number of cycles.

As shown in Fig.2, the LMS can do only one-dimensional scanning of 180 degree. However the aim of the study is to obtain the cargo hold size and shape of the bulk cargo, as shown in Fig. 3.

To use the multi-axis servo motion system to achieve multiple dimensional scanning.

Firstly, the range and scanning interval of the LMS detection are determined. The scanning interval is rotation angle of the mirror within the LMS, and it will emit a laser beam at each rotation angle and obtain a value [8]. So the smaller is the value of the interval, the higher is the detection accuracy. LMS offers a maximum detection range of 180 degrees, and the minimum interval is 0.25 degrees, which is a straight angle rotating from 0 degree, as shown in Fig. 2. When LMS variables are set up, we can outline the outer contour line of the target. When the LMS rotates a circle in the horizontal plane to detect the targets for several times, the approximate shape of the cargo holds and bulk cargo can be generated.

But the 3D points obtained after decoding is in the polar coordinate system (l,a) whose pole is the laser radar detection center, where / is the distance between the LMS and the target, a is the angle between this laser beam and horizontal direction. It is needed to convert the polar coordinate system to a rectangular coordinate system to represent the data in the rectangular coordinate system [9].

Fig. 4 is the geometric relationship between the laser radar and materials.

In the figure, O is the location of the LMS. P is the scanning point and a is the angle between OP and the horizontal direction, 2D coordinates of the scanning point P can be obtained: ...(3) The LMS rotates around the x-axis, forming an angle ß to it. Assuming the laser radar scan line is projected to XOY plane, three-dimensional coordinates of the scan point R with respect to the laser radar is: ...(4) 2) Image Edge Extraction Algorithm of cargo hold section.

As shown in Fig. 5, the LMS is located above the hold, where we want to detect only the shape of the cargo hold. While with more data, the processing speed will be more slowly. So the ideal situation is that the detection range of laser radar can cover just the cargo hold. According to the site observation, selecting the detection range of 180 degrees can fully meet the need.

When it comes to the detection of the shapes of the cargo holds, initially the LMS detection process may produce a number of discrete points. According to the initial setting range of the finder, we get the maximum detectable distance in the current state, so the point beyond the maximum distance will be recognized as an invalid point by the program.

In order to obtain more obvious edge points, this article will use the gradient algorithm to detect the cargo hold edge [10].

The Gaussian function of dimensional image signal is: ...(5) ...(6) ...(7) Then smooth the image[10, 11]: ...(8) The first order derivative in a certain direction N is: ...(9) where n is the direction vector, VG is the gradient vector.

Make a convolution of the image /(x,_y) and Gn, and change the direction of n at the same time. The n is the direction orthogonal to the detected edge when the maximum value of Gn*f(x,y) is obtained.

...(10) ...(11) ...(12) ...(13) A(x,y^ represents the edge strength of the point (x,j)in the image, #is the normal vector of the point (x,_y).

If the modulus of two-dimensional image wavelet transform \Msf(x,y)\ » T and the gradient direction code CodeAsf(x,y) reached modulus maxima, then (x, _y)is a boundary point [12]. Repeat this step until edge recognition is completed.

Using the least squares method to perform the final match [11], you can get the best matching results.

3) Hold hatch pattem matching algorithm.

Since the point data obtained by LMS is generally raw measurement data, so it must be registered and filtered before using to reduce data operations and reduce noise interference.

During hold shape fitting, an edge extraction algorithm was used according to the actual situation. As follows: First, find a point xO right under the center point of the range finder.

Then find the right point xl, obtain the slope K1 of the line segment connecting xl and xO.

Turn to the right and obtain the slope K2 of the line segment connecting x2 and xl. If Kl= K2, it is stated that xO, xl, x2 in the same line. Meanwhile, considering the structure may have some projections bilge and laser measurement errors existing in the detection, set a slope difference allowable range T, and if the absolute difference between K2 and K1 is less than T, it is said it has not reached cargo hold edges.

Following steps 2 and 3, continue to find until the absolute value of the difference between Kn and Kn1 1 greater than T, then it is said the edge points xn has been found.

Following steps 2, 3 and 4, turn left to find the left edge point xn'.

According to this algorithm, we can detect two points on the edge of the hold hatch and get the horizontal distance between the center of the rangefinder and the two points. The sum of the two distances is the length between the two points on the edge of the hatches. Rotating in a horizontal plane, the laser radar scans several times to obtain multiple sets of points on the edge of the cargo hold. The shape of the cargo hold comes out if these points are sequentially connected together by a straight line. As shown in Fig. 6.

3.2. Automatic Extraction Algorithm of Material Shape In the port cargo handling operation, the distribution, the maximum height, minimum height and volume need to be monitored real-timely. The operating parameters of subsequent operations can be determined after calculation in a certain rule.

Hatch image fitting algorithm from the previous section can also be used to find edge points in vertical plane{y0,y{,y2,...,yn} .The points will be converted to a two-dimensional plane using formula (3) and the shape of the bulkhead can be fitted [13, 14], as shown in the Fig. 7.

In bulk operations, continuity of operations is extremely important. Once the process is stopped or shut down, it is possible to cause shovel clogging, resulting in equipment wear and unnecessary manual work, affecting operational efficiency. Therefore, it is necessary to monitor the inclination of the vessel real-timely to prevent overturning incident. The inclination angle can be calculated by the slope K of the edge line and the slope K can be obtained through the least square method, as shown in Equation (14). The inclination angle of the cargo hold angle a can be solved by using Equation (15). The system automatically adjusts the trajectory of blanking according to the inclined angle, and automatically controls the inclined angle to be in 2 to ensure safe operation.

...(14) ...(15) Assuming ... points p.. found on the cargo hold walls, set the threshold T, as shown in Equation (16) [15]: ...(16) If the absolute value of data scanned by LMS is larger than the threshold T, the point cannot be the bulk point. The software will automatically filter out the point whose absolute value is greater than the threshold value T, the leaving points whose absolute value are less than T are the points of the bulk. Make a straight fitting of these points through the least square method and we can get the shape of the material, as shown in Fig. 8.

4. Experimental Results in Tianjin Port After the automation reformation of the ship loader, a shipment experiment was carried out in Tianjin Port Coal Terminal to test the performance of the system. As shown in Fig., the laser radar is installed on the ship loader and it detects the shape of the cargo hold and the location of the material from different places above the cargo hold, as shown in Fig. 10. Experimental results are recorded in Table 1-3.

Based on the results, the error of the cargo hold size and cargo height is controlled within 1 %. The error of the ship inclination angle is controlled within 5 %. The error of the inclination angle is a little bigger, because the ship was always shaking on the water when the experiment was processed. The measured results and the actual results are very close, which can meet the requirements of automated loading operations.

5. Conclusion This paper describes an approach to inspect the cargo hold, the cargo height and the inclination angle by using LMS. The LMS is a novel sensor to accurately measure distances of the target in an arc section. From the 3D points rebuilt by LMS, the target information is recognized, which can assist the ship loader operation automatically. The experimental results show those methods are all valid and enough precise for automation.

As the rapid growth of the world's bulk cargo throughput, bulk ports face a lot of pressure. To build bulk automated production and information management is imperative. The new ship loader automatic mode is a component of the automation system to adapt to this new form of future bulk ports. It can achieve unmanned operation to ensure safe and reliable automated loading and unloading operations. It saves manpower and at the same time improves operational efficiency, can be expected to have broad application prospects in the future.

Acknowledgements This research is supported by "Young University Teachers' Training Project" of Shanghai Education Commission and "Local University Capacity Promotion Special Programs (13510501800)" of Science and Technology Commission of Shanghai Municipality.

References [1]. Bao Qifan, Research and development of automatic bulk cargo equipment in modem ports, China Mechanical Engineering, Issue 23, 2008, pp. 2797-2803.

[2]. Chris Mansour, Ari Steinberg, Neven Matasovic, Analysis, design and construction of the supporting structure and wharf retrofit for a new shiploader at the Port of Long Beach, California, in Proceedings of the Conference Ports 2004: Port Development in the Changing World, 2004, pp. 1049-1057.

[3]. Gao Meijing, Wu Weilong, Bi Weihong, Gu Haihua, Large field of view real-time three-dimensional imaging for ports, in Proceedings of the SPIE - The International Society for Optical Engineering, 2011.

[4]. Daniel J. Johnson, Process and equipment automation for container terminals, in Proceedings of the 12th Triannual International Conference, 2010, pp. 797-806.

[5]. Liu Wei, Xue Shilong, Research on auto control system for ship loader based on AC800M, Advanced Materials Research, Vol. 383-390, No. 2, 2012, pp. 1231-1234.

[6]. Wenjun Zhang, Tao Wu, Remote 3D measurement & visualization system based on laser scanning & 3D reconstruction, in Proceedings of the Symposium on Photonics and Optoelectronic (SOPO), Chengdu, 19-21 June 2010, pp. 1-5.

[7]. Huipu Xu, Zi Ma, Yuqing Chen, A modified calibration technique of camera for 3D laser measurement system, in Proceedings of the IEEE International Conference on Automation and Logistics (ICAL'09), Shenyang, 5-7 August 2009, pp. 794-798.

[8]. Zheng Keqiang, Zhuang Yan, Wang Wei, Automated 3D scenes reconstruction for mobile robots using laser scanning, in Proceedings of the Control and Decision Conference (CCDC '09), Guilin, 7-19 June 2009, pp. 3007-3012.

[9]. Duan Pan, Xie Kaigui, Song Na, Duan Qichang, A method of vehicle license plate de-noising and location in low light level, Journal of Networks, Vol. 5, Issue 12,2010, pp. 1393-1400.

[10]. Yingkun Hou, Chunxia Zhao, Mingxia Liu, Zhengli Zhu, Deyun Yang, Semisubsampled wavelet transform based image watermarking with strong robustness to rotation attacks, Journal of Multimedia, Vol. 5, Issue 4,2010, pp. 385-392.

[11]. Muhammad J. Mirza, Nadeem Anjum, Association of moving objects across visual sensor networks, Journal of Multimedia, Vol. 7, Issue 1, 2012, pp. 2-8.

[12]. Jinyong Cheng, Yihui Liu, 3-D reconstruction of medical image using wavelet transforms and snake model, Journal of Multimedia, Vol. 4, Issue 6, 2009, pp. 427-434.

[13]. Xuesong Mao, D. Inoue, S. Kato, M. Kagami, Amplitude-Modulated laser radar for range and speed measurement in car applications, IEEE Transactions on Intelligent Transportation Systems, Vol. 13, Issue 1,2012, pp. 408-413.

[14]. G. Biegelbauer, M. Vincze, 3D Vision-Guided bore inspection system, in Proceedings of the IEEE International Conference on Computer Vision Systems (ICVS '06), 4-7 January 2006,22 pages.

[15]. Liu Songtao, Lu Bin, Wang Huili, Yin Fuliang, Ship visible image segmentation method based on combining coarse and precise interaction and iterative graph cut, Journal of Optoelectronics Laser, Vol. 23, Issue 8,2012, pp. 1609-1615.

[16]. Mi Chao, Liu Haiwei, Zhao Ning, Shen Yang, A ship cargo hold inspection approach using laser vision systems, Telkomnika, Vol. 11, Issue 1, 2013, pp. 330-337.

1,2 Chao MI,2 Youfang HUANG,3 Haiwei LIU, 4 Yang SHEN,2 Weijian MI 1 School of Economics & Management, Shanghai Maritime University, Shanghai, China 2 Container Supply Chain Tech. Engineering Research Center, Shanghai Maritime University, Shanghai, China 3 Logistics Engineering College, Shanghai Maritime University, Shanghai, China 4 Higher Technical College, Shanghai Maritime University, Shanghai, China E-mail: [email protected] Received: 6 September 2013 /Accepted: 25 October 2013 /Published: 30 November 2013 (c) 2013 International Frequency Sensor Association

[ Back To TMCnet.com's Homepage ]