Multiple Vehicles’ License Plate Tracking and Recognition via Isotropic Dilation

Lorita Angeline1 Wei Leong Khong Farrah Wong Ismail Saad Kenneth Tze Kin Teo2

Modeling, Simulation and Computational Algorithm Laboratory School of Engineering and Information Technology

Universiti Malaysia Sabah Kota Kinabalu, Malaysia

angeline.lorita@gmail.com1 ktkteo@ieee.org2 Abstract — In this paper, a new algorithm for Automatic License Plate Localisation and Recognition (ALPR) is proposed on the basis of isotropic dilation that can be achieved using the binary image Euclidean distance transform. In a blob analysis problem, any two Region of Interest (RoIs) that is discontinuous are typically treated as separate blobs. However, the proposed algorithm combine with Connected Component Analysis (CCA) are coded to seek for RoI within a certain distance of other RoI to be treated as non-unique. This paper investigates the design and implementation of several pre-processing techniques and isotropic dilation algorithm to classify moving vehicles with different backgrounds and varying angles. A multi-layer feed-forward back-propagation Neural Network is used to train the segmented and refined characters. The results obtained can be used for implementation in the vehicle parking management system.

Keywords - isotropic dilation; euclidean distance transform; license plate localisation; blob analysis; overlapping bounding boxes; connected component analysis; image processing

I. INTRODUCTION Automatic license plate localisation and recognition

(ALPR) is an essential subject of research due to its numerous fields of applications. Localisation and recognition of vehicle’s license plate reported in existing research in general composed of several processing steps: extraction of license plate region, segmentation of characters, and recognition of each character [1]. The escalating numbers of vehicles draw a positive implication to provide a professional and efficient parking system. Hence, a license plate number recognition is needed to support the parking management system [2]. As each and every vehicle will carry a unique license plate and there are no external cards, tags or transmitter need to be recognised [3]. Therefore, image processing based algorithm is practicable to develop a vehicle parking management system. Thus, the aim of this project is to design a system to detect and recognise the license plate of a moving vehicle.

The rest of the paper is prepared as follows. The next section makes up of a review of other related and similar researches that have been covered in the literature. An overview of image processing approach and algorithms is given in Section III. Experimental results, limitations and discussions will be delivered in Section IV to demonstrate feasibility of the system. Overall project and future extensions are concluded in Section V.

II. REVIEW OF OTHER RELATED WORKS The main purpose of this section is to render a brief

reference source for the researchers involved in license plate localisation, despite of any specific application fields (for instance, parking management system, traffic surveillance, optical character recognition (OCR) and etc.).

Sobel operator and edge detector techniques are proposed in many literatures for vehicle’s license plate detection and localisation. Edges in a mix image have been widely used as an important criterion to detect and locate a vehicle’s license plate. A license plate with frame can be easily located by using this technique, but it cannot be implemented to a frameless license plate. Without a good frame of a license plate, the localised and flood-filled edges unable to form a proper rectangular to be processed in the next step. As a result, the coordinates of the license plate cannot be determined [4].

Some researchers utilised the smearing algorithm to overcome this restriction. This algorithm is used to detect and locate the license plate on a vehicle by detecting the spaces in between each character. A grey scale image is processed along a horizontal scan. Two threshold values need to be set beforehand. If the numbers of white pixels are less than a desired or greater than the second desired threshold, the white pixels will be converted to black. Reference [3] suggested the threshold values for both horizontal and vertical smearing are preferable to be set as 10 and 100. Then, image will be further processed in vertical scan.

A novel texture descriptor is based on line-segment features for text detection in images and video sequences. Reference [5] suggested a descriptor based on Improved Connective Hough Transform (ICHT), which represents a perceptual characteristic of texture, in terms of directionality, regularity, similarity, alignment and connectivity. A dense vertical edges region (text-like) is extracted from the contrast enhanced gray-scale image. This is handled via fast line-segments based processing and a simple geometrical filter. Next, an invariant descriptor is used for training and verifying the correctness of the detected regions.

Character segmentation is an important step in vehicle’s license plate localisation and recognition system. It is a process that separates words to a single character for easy identification [6]. There are numbers of difficulties in this step, such as influence of image’s noise, frame of the license

2011 Third International Conference on Computational Intelligence, Communication Systems and Networks

978-0-7695-4482-3/11 $26.00 © 2011 IEEE

DOI 10.1109/CICSyN.2011.24

54

plate, rivet, space mark, vertical overlap, horizontal overlap ligature, two-connected characters and etc.

Reference [7] proposed a new algorithm for character segmentation of a vehicle’s license plate. The algorithm applied Hough Transformation approach and the prior knowledge in horizontal and vertical segmentation respectively to solve the difficulties mentioned previously. The information of intensity is used to avoid the abruption and conglutination of characters which are the primary drawbacks from a binary image.

There is another approach where Hough Transformation is used to detect lines in an image [8]. A curve r, is obtained for each pixel in an image space. Suppose that there are n points exist in the image space, then n curves will be obtained in the parameter space. If any of these curves cross the same point, then the n points in the image space will be determined as on a line. Therefore, a line in a space can be found by searching the cross points in the parameter space.

The algorithm suggested has several advantages. Firstly, Hough Transformation implementing a vote strategy, and the invalid subsection lines are set as the minority so that it can be eliminated. Second, this algorithm based on local projection technique, which the effect from the background, illumination variance and rotation of the license plate can be weaken. This approach also avoids the necessity to rotate the image for rectification.

Reference [9] proposed another method for character segmentation using horizontal scan line and clustering approach. The segmentation involves a process where collection of filtered character is distinguished to a sequence of characters that will be used in the later processing stage. This step is very essential due to the overlapping characters that are formed in the license plate image. Clustering technique is applied to identify significant blobs. Initially, the mixed image is enhanced using simple image enhancement techniques such as Fixed Filter, Minimum Filter, Opening and dependent threshold. Then, the image will be further processed using horizontal scan line and clustering technique. Thoroughly each image will be transformed into blob objects. Information such as location, height, and width of the license plate are analyzed and used in cluster exercising and choosing the best cluster with winner blobs.

Reference [10] proposed a novel character classification and recognition algorithm. The system comprises an interface between the PC and the real-time DSP based Video Image Processing System that is used for capturing car plate images. There are three primary steps in pre-processing of the captured car plate image. These primary steps are brightness correction, contrast correction and noise filtering. A database that provides information about each character is created by studying various combinations such as where the points are linked together and number of such points that exist. The advantage of this technique is that it can be implemented using the real-time digital signal processing and returns a real-time recognition result.

Template matching has been one of the widely used tools for character recognition. This technique has the advantage for recognition of the single font, not rotated, not tilted, and fixed size characters. Although this approach preferably utilised with binary images, but a grey level images also can obtain pretty good results from a properly built templates. Template matching method must have character images as template stored in memory. The segmented character will be identified by calculating the correlation coefficient. The idea behind the implementation of a correlation based identification scheme is when two template pools, one consisting of all the possible values of letters, and of all values of the digits, are constructed.

III. METHODOLOGY An overview of image processing techniques used in this

paper is summarised as in Fig. 1.

A. Motion Detection and Pre-Processing Pre-processing is the requisite step to be taken for any

image analyzing program. Without a proper pre-processing phase, the later segmentation and recognition procedure could be very ineffectual. It is very crucial to perform image enhancement to highlight the desired region in the processed image. Therein, it will be the vehicle and its license plate.

Fig. 2 shows image frames fed to the system. Background subtraction is performed on the image frames and follows by essential image enhancement steps: thresholding and median filtering. The algorithm of the motion detection and tracking of a moving vehicle are based on the authors’ previous work [11].

B. Isotropic Dilation combine with Connected Component Analysis(CCA) In a blob analysis problem, any two RoIs that are

discontinuous are typically treated as separate blobs as shown in Fig. 3 and 4. To overcome this limitation, an

Figure 1. Overview of image processing techniques.

55

Figure 2. Motion detection and pre-processing.

analysis needs to be done on finding out which pixels are within a certain distance of the foreground in a binary image. A useful technique called isotropic dilation that can be achieved using the binary image Euclidean distance transform has been studied in this paper. This technique combine with Connected Component Analysis (CCA) are coded to search for RoI within a certain distance of other RoI to be treated as non-unique. This approach is the main core for two cases in this paper which the first one is to distinguish multiple moving vehicles, and the second case is to locate license plate region.

After the image frame is processed using simple enhancement technique, Euclidean distance transform is computed for each pixel to its nearest foreground pixel. For each pixel in the image, the distance transform assigns a number that is the distance between that pixel ( 1x , 1y ) and the nearest nonzero pixel ( 2x , 2y ) of the image as in (1). In case 1, the distance between the two pixels is set to 16 pixels, whereas in case 2, the distance is set as 4 pixels. This implies that for any two RoIs that discontinuous and having a distance of 16 pixels (case 1) will be treated as non-unique. Similarly, any two RoIs that discontinuous and having a distance of 4 pixels will be treated as non-unique (case 2). Fig. 5 and 6 show the isotropic dilation result for case 1 and 2.

221

221 )()( yyxx −+− (1)

Next CCA analysis is performed on the resulted images. However, the isotropic dilation step has caused the RoIs padded with too many pixels. Therefore, all the elements in the resulted image are set to “0” corresponding to the background pixel in the input image. Finally, area of labelled components are analysed to find the valid RoI. Fig. 7 shows Signature Analysis as in [11] is performed to locate the license plate. Table I presents the pseudocode of isotropic dilation combine with CCA.

Referring to the diagram in Fig. 8(a), two equations can be derived to determine if the two bounding boxes are overlapped. The first condition (1) is that if the radius from

C. Distinguishing Overlapping Bounding Boxes The main issue in tracking and distinguishing two

moving objects is when these two objects come too close to each other and cause false tracking and localisation.

The proposed algorithm is further analysed to overcome this limitation. One method to go about this is to store the bounding box properties and its size to be used as reference in the next iteration. Thus a subroutine is added to the algorithm to check for overlapping of bounding boxes. For this solution, the bounding box need to be broke into some key components. A bounding box has a centre point and two significant radiuses. A radius from the centre point to the edge in the x- direction, and a radius from the centre point to the edge in the y-direction.

Figure 3. A typical blob analysis for case 1.

Figure 4. A typical blob analysis for case 2.

Figure 5. Isotropic dilation for case 1.

Figure 6. Isotropic dilation for case 2.

56

TABLE I. ISOTROPIC DILATION COMBINE WITH CCA PSEUDOCODE

Input: image frame difference 1 perform isotropic dilation {

if case 1: Distinguish multiple moving vehicles

img= bwdist(img)<=8 if case 2: License plate region img= bwdist(img)<=2 }

2 perform Connected Component labeling for i=1:last_component

label the "fat-padded" components result will be grouped of RoIs treated as non-unique

3 eliminate unnecessary padded pixels 4 measure properties of image region 5 find area>50 pixels

output: case 1: RoIs (distinguished vehicles)

case 2: RoI (license plate region)

Figure 7. Signature Analysis to locate license plate region.

The centre of bounding box 1 to bounding box 2 ( xr12 ) is less than or equal to the sum of the radius of 1 in the x-direction ( xr1 ) and the radius of 2 in the x-direction ( xr2 ); then bounding boxes 1 and 2 are coordinated such that they share common coordinates in the x-plane. Equation (2) is formulated by using the same derivation for the y-plane.

+≤ xx rr 112 (2)

+≤ yy rr 112 (3)

Next, the sizes of the bounding boxes in the previous iteration are compared with the sizes in the current iteration. An analysis is carried out to check for pixels contribution from one bounding boxes to the others. Fig. 8(b) shows sizes of the bounding boxes for two consecutive bounding boxes. The figure implies that some white pixels in bounding box 2 are contributed to bounding box 1 due to overlapping of the two bounding boxes.

D. Character Segmentation and Recognition Following the successful vehicle’s license plate

localisation in image frames, procedures such as character segmentation, thinning algorithm and determining of universe of discourse are performed. There are several widely used techniques such as vertical projection, feature

Figure 8. Distinguishing two overlapping bounding boxes.

vector extraction, mask technique and CCA to perform character segmentation.

To simplify the process of identifying the characters, it is preferable to split the extracted plate into several images. Therein, each character will be classified as a connected component calculating the standard deviation of columns and rows and finally normalisation of each. CCA is implemented in this project on account of its simplicity works on both binary and grey level image and different measures of connectivity are doable. After all the characters have been successfully segmented, it will be inverted so that the character is in black colour instead of white. This is due to the characters used for neural network training are in black colour.

Next, thinning algorithm is performed on the segmented character before it is fed to the neural network for testing. This algorithm removes the pixels so that an object without holes shrinks to a minimally connected stroke, which a one pixel width ‘skeleton’. On the other hand, an object with holes shrinks to a connected ring halfway between each hole and the outer boundary.

The result has several advantages; it preserves the Euler number, retains the topology of the original object, and forces the ‘skeleton’ being in the middle of the segmented character. Besides, the thinning approach also preserves the shape of the image, thus makes it a significant feature desirable for object recognition or extracted the image for classification. In detecting and tracking a moving vehicle’s license plate, angles and distance from the camera influenced the quality of the extracted license plate; as the width of the characters may be differ from one and another. Thinning algorithm can be very beneficial here as the different width of each character is not the main issue anymore.

The size of the segmented-thinned characters is not fixed since they are exposed to noises such as illumination change, intensities, shadow and etc. These noises tend to affect the cropping process to be varied from one to another. This caused the input of the network become nonstandard and prevents the data from feeding through the network. Universe of discourse is defined as the shortest matrix that

(a) Overlapping edges

(b) Searching for pixels contributionFrame 40th Frame 50th

57

Figure 9. Character segmentation and recognition procedures.

fits the entire character skeleton. The universe of discourse is performed on the segmented-thinned character in order to standardise it. The image pre-processing is then followed by image resize to meet the network input requirement before fed to the neural network for both training and recognition procedures.

In this paper, a simple multi-layer feed-forward back-propagation Neural Network with 35 inputs neuron is used to recognize the segmented characters. Early pre-processing steps performed on the segmented characters play a crucial role in determining the feasibility of the recognition. The recognised characters from the license plate will be stored in the database. Fig. 9 shows the segmented-thinned characters and database for the Artificial Neural Network system.

IV. EXPERIMENTALS RESULTS AND DISCUSSIONS The proposed system has been simulated using Matlab

7.10 and MySQL Community Server Version 5.1.29. The architecture of this simulation test involves an Intel Core 2 Duo CPU and 2GB of server memory. There is no specific limitation for the camera. The prototype system works well for a clear image video taken from a variety of camera resolutions (minima 2 mega pixels). The dimension of the sample video is 640 x 480 pixels. The distance between the camera setup and the vehicle in motion varied from 2 to 10 meters at a height of 1.5-1.8 meters from the ground. The system was simulated with vehicles in motion with different backgrounds, under illumination changes and varying poses and angles.

The major difficulty in tracking and localisation of moving vehicles is the low quality input during the acquisition stage. The environment of the acquisition stage such as illumination, insufficient of light level and incorrect auto focus of the camera are among the factors. Fig. 10 shows the process of localisation of moving vehicle under insufficient light level. In the first iteration, the moving vehicle is detected. However as the vehicle moving forward,

the system failed to detect the whole vehicle due to insufficient of light to make it visible to the system.

Besides, the direction of sunlight is constantly changing as the day progresses and those changes may lead to misdetection as shadows occur following the moving objects. Image taken from outdoor parking spaces and indoor parking spaces are influenced by the sunlight and fluorescent light. Fig. 11 illustrates the effect of shadow in an outdoor environment. The input is taken where a vehicle is going down a slope. From the first iteration until the fourth iteration, the system is unable to distinguish the shadow and the moving vehicles.

Another issue in the license plate segmentation is the effect of intensities. The surrounding light intensities might cause reflection on the license plate and lead to distortion in the next processing stage.

Figure 10. Insufficient of light level.

Figure 11. Shadow effect.

(a) Input (b) 1st Iteration (c) 4th Iteration

(a) Segmented character

(b) Thinned character (c) Universe of discourse

(d) Noise-free characters (e) Samples of noisy license plate

58

Fig. 12 shows the successful results in tracking and localisation of two moving vehicles and the results are stored in the database. For overlapping bounding boxes analysis, at frame 40th, it is noticeable that the two vehicles still treated as separated blobs. As vehicle2 getting closer to vehicle1 (at frame 50th), some white pixels in vehicle2 enter the set Euclidean distance transform of vehicle1 and treated as non-unique.

Three conditions need to be satisfied in order to identify if overlapping of objects is occurred. Once all the three conditions are satisfied; both (1) and (2) are true, and pixels contribution is occurred, it can be concluded that the two bounding boxes must be overlapped or share a common edge. At frame 90th, parts of vehicle2 disappear behind the vehicle1. At this iteration, both vehicles still treated as separated. Information on bounding boxes properties and its size in the previous iteration is pulled out to calculate the actual blob’s size of the two vehicles. At frame 115th, vehicle2 is totally blocked by vehicle1.

V. CONCLUSIONS The aim of this paper is to experiment the potential of

designing a system for vehicle motion detection and license plate localisation under non-ideal environment such as the existence of other objects that leads to unsuccessful results. A new algorithm based on isotropic dilation technique combined with Connected Component Analysis is coded to look for RoI within a certain distance of other RoI to be treated as non-unique. The proposed algorithm is further refined to distinguish overlapping of bounding boxes. Next, the localised license plate region is segmented to isolate each of the character. Finally, the isolated character is thinned before it is fed to the Artificial Neural Network system for recognition purpose and stored in the database.

Based on the experimental results and discussion, it can be observed that the proposed system needs to be improved to be more robust. The major concern in this study is the environmental factors such as the shadow effect and insufficient of light level. The system still needs to be enhanced in the future to overcome the mentioned restrictions.

ACKNOWLEDGMENT The authors would like to acknowledge the funding

assistance of the Universiti Malaysia Sabah under the postgraduate research grant, GPS0001-TK-1/2009 and University Postgraduate Research Scholarship Scheme (PGD) by Ministry of Science, Technology and Innovation of Malaysia (MOSTI).

REFERENCES [1] Xiaodan Jia, Xinnian Wang, Wenju Li, Haijiao Wang, “A novel

algorithm for character segmentation of degraded license plate based on prior knowledge”, Proceedings of the IEEE International Conference on Automation and Logistics. p249-253, 2007.

[2] Haris Al-Qodri Maarif and Sar Sardy, “Plate number recognition by using neural network”, University of AL-Azhar Indonesia. Prosiding Semiloka Teknologi Simulasi dan Komputasi serta Aplikasi 2006. p176-182, 2006.

[3] Serkan Ozbay and Ergun Ercelebi, “Automatic vehicle identification by plate recognition”, World Academic of Science, Engineering and Technology, 2005.

[4] L.Angeline, K.T.K. Teo and Farrah Wong, “Smearing algorithm for vehicle parking system”, Proceeding of SEIT 2009 [Kota Kinabalu, Malaysia], Proceedings of the 2nd Seminar on Engineering and Information Technology, July 2009. pp. 331-337.

[5] Chu Duc Nguyen, Mohsen Ardabilian and Liming Chen, “Robust car license plate localization using a novel texture descriptor”, 2009 Advanced Video and Signal Based Surveillance. Proceedings. IEEE. pp. 523-528.

[6] B. Al-Badr and S.A.Mahmoud, “Survey and bibiliography of arabic optical test recognition”, Signal Processing, 1995.

[7] Yungang Zhang, Changsui Zhang, “A new algorithm for character segmentation of license plate”, Intelligent Vehicles Symposium, 2003 Proceedings. IEEE. pp. 106-109

[8] Yuan Jie, Hu Zhengyi, Wang Yanping, “Finding corners by using Hough transform”, Wuhan University Journal of Natural Science, Jan 1998.

[9] Siri Norul Huda Sheikh Abdullah, Marzuki Khalid and Rubiah Yusof, “License plate recognition using multi-cluster and multilayer neural networks”, 2006, Proceedings, IEEE. pp. 1818-1823.

[10] B. Lim, W. Yeo, K. Tan and C. Y. Teo, “DSP based real time character classification and recognition for car plate dectection and recognition,” IEEE Conference on In-Circuit Serial Programming (ICSP), pp. 1269-1272, 1998.

[11] L.Angeline, M.Y. Choong, Farrah Wong and K.T.K. Teo, “Tracking and localisation of moving vehicle license plate via Signature Analysis”, Proceeding of ICOM 2011, 4th International Conference on Mechatronics, May 2011.

Figure 12. Successful results.

59