Keywords

1 Introduction

Crop growth information is the basis of precise management of crop production, which plays a decisive role in the management of growth, quality and yield of crops. Leaf information is a direct reflection of crop growth status, and it is a research focus to extract leaf region effectively from crop image in the present research [1, 2]. Cotton is strategic materials relating to the national economy and people’s livelihood, and leaf is also an important organ for photosynthesis of cotton. The size of leaf area has a direct effect on the yield of cotton in a certain extent. Therefore, establishing a convenient and accurate method for obtaining leaf area is of positive significance to guide cotton production time activity and develop high yield, high quality and high efficiency cultivation technique measures [3, 4]. In the process of agricultural automation, machine vision technology has become an indispensable part. This technology has been used in many fields of agricultural automation to mine the data from crop images, such as crop water stress [5, 6] and detection of crop diseases [7], etc. In recent years, many researchers have developed leaf separation algorithms to separate leaves from the crop images by machine vision [8, 9], and these algorithms have been applied to crop identification, weed control, and some other fields. However, there are different degrees of over segmentation and under segmentation in the extraction of leaves with shadow and overlap**. Therefore, in order to meet the requirements of the practical application, it is needed to develop an effective algorithm to extract the area of overlap** and shaded leaves. The targets of this paper were to extract the single cotton leaf quickly and accurately in cotton images obtained in natural light condition, and to compute the leaf area size to provide data reference for crop growth monitoring [10].

The extraction of cotton leaf area based on machine vision technology will generally use the image segmentation. The basic idea of the threshold method is to calculate one or more gray thresholds based on the gray level of the image, and the gray value of each pixel in the image is compared with the threshold value, the pixels are classified according to the comparison result [11–13]. The difficulty of this algorithm is to calculate the optimal segmentation threshold value and the segmentation effect is uneasy to grasp. In image segmentation, image edge information is often used. The edge is a collection of two different regions of the boundary line of the image, which is a reflection of the discontinuity of the local feature of the image and the change of image features, such as gray, color, texture, and so on [14–16]. However, the traditional edge detection is not ideal for the detection of overlap** edges. Lifting wavelet transform not only has good time-frequency local characteristic and multi-resolution analysis characteristic and avoids the loss of information due to the limitation of the computer accuracy when dealing with the image. Canny operator can detect the real weak edge, and it is the optimal edge detection operator. Researchers have used the combination of them to get a good edge detection effect [17], but they did not have a further image segmentation.

Watershed algorithm [18] is a morphological segmentation algorithm, which is based on the gradient of the image. The weak edge can be disposed effectively with this method, but the noise in the image can cause over segmentation of the watershed algorithm. So the traditional watershed segmentation algorithm is not fit for direct application. Therefore, many researchers have proposed the improvement methods for the traditional watershed algorithm [9, 19].

Major difficulty of the extraction of single leaf area in natural light is to handle the cotton leaf images with occlusion and shadows. A method based on lifted wavelet and improved watershed method was presented to extract cotton single leaf in this paper.

2 Experiments and Methods

2.1 Experimental Material

In laboratory conditions, images were collected in Bei**g Academy of Agricultural and Forestry Sciences. In order to simulate the natural conditions of the day, the cotton plants were placed in the laboratory by the window to receive natural light irradiation, the laboratory temperature was 21 °C, humidity was 39 %. A CMOS camera (Nikon Inc., J1, Japan) with a resolution of 3872 × 2592 was used to get the RGB image, which was saved as a JPG format. According to the number of cotton leaves, cotton images were divided into two groups, images with 2 leaves and images with 4 leaves. According to whether there was occlusion or shadow between cotton leaves in the cotton image, the cotton images was further divided into four kinds. In this paper, Matlab (R2010b) and photoshop 7 software were applied to process the image. In order to improve the processing speed, the cotton RGB image was reduced to one fifth of the original image size, which was 510 × 775.

2.2 Experimental Methods

2.2.1 Overall Description of the Algorithm

Firstly, the green area of cotton in an RGB image was extracted based on color threshold and filtered the region that were not interested in the image, including most of the shadows, background and cotton stalk, in this way the cotton leaf area could be get. Then lifting wavelet and Canny operator for respectively accomplished the image gray enhancement and edge detection so that image edge could be effectively and accurately extracted from occlusive leaves, based on the operations above, watershed algorithm could achieve better segmentation effect. Finally, in order to solve the over segmentation caused by directly using the watershed transform, the leaf area image was converted to HSV space and morphological marking of the foreground and background was processed in the image to improve the watershed algorithm to reach a better effect of segmentation.

2.2.2 Extraction of Leaf Area in the Image Based on the Color Threshold, Lifting Wavelet Transform and Canny Edge Detection Operator

Color threshold (G−R)/(G+R) applied to highlight the green component in the cotton image, the threshold value of the component was automatically calculated by the maximum difference method. Wavelet transform could enhance the gray image, but in practical applications, due to the limitation of computer calculation precision, the image would produce information loss after wavelet transform, lifting wavelet overcome the shortcomings. Canny operator was the most effective edge detection operator of edge function. After applying the lifting wavelet and Canny operator, the gray level of the image could be effectively enhanced. Finally, the leaf region was extracted from the enhanced edge, which allowed us to use the watershed algorithm obtain a good segmentation effect.

2.2.3 Cotton Image Segmentation Based on Improved Watershed Algorithm

2.2.3.1 2.2.3.1 Principle of Watershed Algorithm

The ideological source of the watershed algorithm [18] is in topography, the topography image is seen as a natural landscape covered by water, the gray value of each pixel in the image indicates the altitude of the point, each of its local minimum and its influence area is called catchment basin, two adjacent catchment basin boundary is the watershed. The main process of the algorithm is to find the catchment basin and the algorithm is described as follows.

$$ T[n] = \{ (s,t)|f(s,t) < n\} $$
(2.1)

where,\( f(x,y) \) is image, \( T[n] \) represents a collection of coordinate \( [s,t] \), the points in the collection are located below the level of the \( f(x,y) = n \).

$$ C_{\text{n}} (M_{i} ) = C(M_{i} ) \cap T(n) $$
(2.2)

where, \( C_{n} (M_{i} ) \) can be seen as the two value image given by formula 2.2, \( \left\{ {{\text{M}}_{ 1} , {\text{M}}_{ 2} ,\ldots \ldots {\text{M}}_{\text{n}} } \right\} \) is the coordinate of the local minimum point of \( f(x,y) \), set \( C[M_{i} ] \) as the coordinate of a point, this point is located in the catchment basin associated with the local minimum in the \( M_{i} \).

$$ C[n] = \bigcup\limits_{i = 1}^{n} {C_{n} (M_{i} )} $$
(2.3)

where, \( C[n] \) is the collection of the catchment basin of the flooded part at the stage \( {\text{n}} \).

$$ C[{ \hbox{max} } + 1] = \bigcup\limits_{i = 1}^{n} {C_{n} (M_{i} )} $$
(2.4)

where, \( C[\hbox{max} + 1] \) is the collection of all the catchment basin, \( \hbox{min} \) and \( \hbox{max} \) represents the minimum value and the maximum value of \( f(x,y) \), as the water level in whole number is increasing from \( {\text{n}} = \hbox{min} + 1 \) to \( n = \hbox{max} + 1 \), the image of the terrain will diffuse through the water.

From analyses above, the result is that every connected component in \( C[n - 1] \) is a connected component of \( T[n] \).

Set \( C[\hbox{min} + 1] = T[\hbox{min} + 1] \) in search of the dividing line, then process the recursive call to obtain \( C[n] \) according to the \( C[n - 1] \).

The watershed algorithm can achieve better results when the targets is connected each other, however, the watershed algorithm is highly sensitive to the change of the image, and the image contains noise and other factors often lead to over segmentation, so the contour of the hope is not covered by a large number of irrelevant contours. The direct application of the watershed segmentation algorithm is not good, so it is necessary to do the pre-processing of the input image.

2.2.3.2 2.2.3.2 Image Pre-processing Before Using Watershed Algorithm

Morphological marker and watershed algorithm are combined to segment the cotton leaf and get the accurate segmentation results. Foreground markers are the dot pixels that are attached to each object and they will create a maximum of each object and find out the target region by these maximums, laying the foundation for the correct segmentation of the image. Background markers are pixels that are not belonging to any object. The method of threshold is used to distinguish the background and the target region to segment images accurately.

The gray scale dilation of image \( f \) with structure element b is called \( f \oplus b \) and described as follows.

$$ (f \oplus b)(x,y) = \hbox{max} \{ f(x - x^{\prime},y - y^{\prime}) + b(x^{\prime},y^{\prime})\left| {(x^{\prime},y^{\prime})} \right. \in D_{b} \} $$
(2.5)

where \( D_{b} \) is the definition domain of b, \( f(x,y) \) is \( { - }\infty \) out of the definition domain of \( f \).

The gray scale erosion of image \( f \) with structure element b is called \( f\Theta b \) and described as follows.

$$ (f\Theta b)(x,y) = \hbox{min} \{ f(x + x^{\prime},y + y^{\prime}) - b(x^{\prime},y^{\prime})|(x^{\prime},y^{\prime}) \in D_{b} \} $$
(2.6)

where \( D_{b} \) is the definition domain of b, \( f(x,y) \) is \( + \infty \) out of the definition domain of \( f \).

The morphological opening operation of image \( f \) with structure element b is called \( f \bullet b \) and described as follows.

$$ f \bullet {\text{b}} = (f\Theta b) \oplus b $$
(2.7)

Morphological opening operation completely removes the object region which didn’t contain structural elements, smooths the object contour, disconnects the narrow connection and removes the tiny protruding part.

The morphological closing operation of image \( f \) with structure element b is called \( f \bullet b \) and described as follows.

$$ f \bullet b = (f \oplus b)\Theta b $$
(2.8)

Morphological closing operation will connect the narrow the gap to form slender curved mouth, and filling the hole with smaller structure elements.

After morphological opening and closing, the image is reconstructed by morphological reconstruction to modify the image and find a maximum value of the unit within each object.

After the operation above, leaf region of gray image becomes smooth, then the revised local extreme value and the image labeled were combined with the watershed algorithm to find the catchment basin and watershed ridge line to realize the algorithm of image segmentation correctly.

2.2.3.3 2.2.3.3 Image Processing Based on Improved Watershed Algorithm

For the sake of objectively evaluating the effect of the method, the effect of automatic segmentation and manual segmentation was compared. After the use of watershed algorithm on the image of cotton leaf, region labeling method was applied for every cotton leaf, then the area of each leaf was found by using the region property. Photoshop 7 software was applied to extract a single leaf and obtain the pixel area. By comparing the two area of the corresponding single leaf, the accuracy of our experimental algorithm could be verified.

3 Results and Discussion

3.1 Single Sample Experimental Results Exhibition

To demonstrate the performance of key steps of the proposed algorithm, an image taken for a four-leaf cotton with partial occlusion was processed as an example. The color threshold of the original image was combined with the lifting wavelet and Canny operator to extract the region of the leaf, the effect was shown in Fig. 1, Fig. 1a is original image, Fig. 1b extracts the leaf region based on the color threshold, lifting wavelet and Canny operator.

Fig. 1.
figure 1

Extraction of leaf area based on color threshold, lifting wavelet and Canny operator (Color figure online)

Full size image

After the extraction of leaf area and its conversion to the HSV color space, the HSV image was transformed to gray scale image. The horizontal and vertical direction of the filtering of the image were carried out by using Sobel edge operator to obtain the modulus. Then the traditional watershed algorithm was applied to edge image. Figure 2a: leaf image is converted to HSV color space. Figure 2b is gradient mode value of HSV image, Fig. 2c is the result of the direct use of the watershed method based on the gradient mode image. It could be seen that direct application of the watershed algorithm would lead to over segmentation phenomenon.

Fig. 2.
figure 2

Direct application of watershed segmentation algorithm (Color figure online)

Full size image

In order to solve the over segmentation problem, morphological marker for images was labeled before using watershed algorithm. Firstly, foreground markers must be connected to the foreground object. Morphological techniques based on the open and close reconstruction were applied to clean up the image. After opening operation, followed by closing operation, the darker spots and stem markers could be removed, treatment effect is shown in Fig. 3, Fig. 3a, b are effect of open operation and reconstructed operation; Fig. 3c, d is the image of the closed operation and reconstruction based on the open reconstruction image. Opening operation could filter out the thrusting which was smaller than structure elements, cut thin lap and played a separate role. Closing operation could fill the gap smaller than the structural elements, lapped short intervals and played a connect role. The reconstruction operation was to clean the image to find flat maximum in each object.

Fig. 3.
figure 3

Morphological markers foreground image

Full size image

Then, the local maximum of the image was calculated based on the open and close reconstruction to obtain better foreground markers. Processing results are shown in Fig. 4. Figure 4a is the region extreme value based on opening and closing operation, which is added to the gray image in Fig. 4b. Figure 4c modifies local maximum values and removes the flecks. Figure 4d applies the threshold method to distinguish the foreground and background.

Fig. 4.
figure 4

Better foreground markers through regional extreme value

Full size image

In the image after the open and close reconstruction operations, the background pixels were marked as black area. The calculation value of two Euclidean matrix of binary image was calculated. For each pixel of the binary image, the distance transform is specified by the distance of pixel and the nearest BW nonzero pixel of the binary image. Then of the watershed ridge line was found. Finally, the image segmentation was implemented based on watershed algorithm, and the results are shown in Fig. 5a. Then the marker matrix is converted into a pseudo color image and shown in Fig. 5b. The pseudo color image is superimposed onto the HSV image and shown in Fig. 5c.

Fig. 5.
figure 5

Image segmentation of watershed algorithm and converted it into pseudo-color image (Color figure online)

Full size image

4 Overall Experimental Results

In order to illustrate the test results, the image of 6 cotton images were numbered in 6 conditions, as shown in Fig. 6. I: 2 leaf of cotton image without shadow; II: 2 leaf of cotton image with shadow; III: 4 leaf of cotton image without overlap and shadow; IV: 4 leaf of cotton image without occlusion and with shadow. V: 4 leaf of cotton image with occlusion and without shadow; VI: 4 leaf of cotton image with occlusion and shadow. The algorithm of this paper was applied to process these images and the segmentation effect are shown in Fig. 6. It is obvious that the 6 images had been correctly segmented. Figure 6a represents the original image, Fig. 6b represents the markers and the target boundary, Fig. 6c represents the pseudo color marker image after segmentation.

Fig. 6.
figure 6

Image segmentation effect (Color figure online)

Full size image

Table 1 compares automatic segmentation with manual segmentation results. The extracted single cotton leaf area was numbered in a clockwise direction. It could be seen that the algorithm in this paper could successfully segment cotton leaves under various conditions and the average correct rate of leaf segmentation was higher than 98 %.

Table 1. Leaf area results of cotton image segmentation
Full size table

5 Conclusions

In order to realize automatic monitoring of the cotton growth status, it is necessary to segment and extract the cotton leaf area. In this process, the segmentation of the cotton leaves with partial occlusion is a key step. This paper presented an algorithm to segment cotton leaf images based on lifting wavelet and improved watershed method. Firstly, the image was segmented by applying color threshold (G−R)/(G+R). Then the region of the leaf and the edge was extracted by the lifting wavelet and Canny edge detection. Through using morphological method, the foreground and background of the image were marked to improve the traditional watershed segmentation algorithm. The leaf extraction accuracy was greatly prompted and the image was segmented successfully. Experimental results on six cases of cotton images showed that the algorithm in this paper was able to successfully deal with the presence of shadow and occlusion in the cotton images, and the single leaf was successfully segmented. By comparing with the results of manually obtained leaf area, the correct segmentation rate of single leaf area was higher than 98 %, which had met the requirements for the single leaf area extraction in practical application. This approach could be used as a preliminary step to build a monitoring system and monitor the growth status of other natural objects, such as wheat, corn, or any other crops or vegetable.