Agricultura 3: 6-11 (2004) Application of image analysis for monitoring growth and development of apple fruits 'Malus domestica' Borkh. during the growing season Denis STAJNKO'* and Miran LAKOTA' 'University of Maribor, Faculty of Agriculture, Vrbanska 30, 2000 Maribor, Slovenia A new approach for counting apple fruits, measuring fruit's diameter and estimating the current yield under flash lighting conditions in the fruit tree plantation was developed and tested in the 2002 and 2003. During the vegetation images often trees were captured five times in both years by applying CCD camera. A close correlation was established between manually counted number of fruits per tree and the estimated number of fruits (r=0.70 to 0.88). However, relatively lower coefficient was estimated for measuring the fruit's diameter (r=0.33 to 0.88). The established correlation coefficients for the average yield per tree was also increasing with the ripening of fruit significantly (r=0.28 to 0.87), therefore the developed algorithm promises a good possibility for forecasting the yield at harvesting on the basis of June and July samples. Keywords: image analysis, apple, Malus domestica, yield, fruit, diameter INTRODUCTION Forecasting the number of fruit and size at harvesting represents together with ecological, varietal and plantation parameters the basis for prediction of future yield and planning of incomes (Weite 1990). In the late decades a considerable research has been conducted in order to develop a viable method for the apple yield prediction, however, today the 'Prognosfruit' Forecast model, developed and introduced in practice by Bavendorf Research Station (Winter 1986), is the only method for yield quantity and quality estimation accepted by the European apple and pear producers. The model is based on the yield capacity of the observed growing unit (trees, variety, rootstock, orchard age and inclination, area), the fruit-set density of the growing unit in the given year and the average fruit mass at a harvesting date (Winter 1986). The accuracy of the method prediction lies between 97 to 98% of future yield for large growing areas with similar environmental conditions (zTrento' in Italy or 'Lake Constance' in Germany) or for estimating average yield for whole countries (Lambrechts 2001). However, as seen from the Prognosfruit annual report 2001, 2002 differences between the forecast and harvested yield have varied from-21.9% to +14.1% per hectare in 2000, depending on the apple variety and country growing region (Lambrechts 2001; Ramos and Lieberz 2003). Despite of the proven efficiency, the main disadvantage correspondence to: University of Maribor, Faculty of Agriculture, Vrbanska 30, 2000 Maribor, Slovenia Tel: ++386 2 25 05 800; fax: ++386 2 229 60 71 e-mail: denis.stajnko@uni-mb.si of the method represents the time-consuming counting measurements of required parameters, which do not allow to predict the future yield in every individual plantation. Possible enhancement of such forecasting methods can be achieved by applying visual techniques in collecting of samples. Nowadays, in fruit growing, vision algorithms are often used for detecting fruits by harvesting machines in experimental cases, but only under controlled lighting conditions (Jimenez et al. 1999). In the last two decades, numerous researches have been involved in the application of image analysis for detecting fruits and guiding the robot hands of autonomous harvesters. As reported by Grand D'Esnon et al. (1987) it was possible to detect different varieties of apple fruits when a protective coverage got a dark background with the first version of apple harvesting robot 'MAGALF. Also Juste and Sevilla (1991) created the citrus robot for harvesting oranges, which required two flashlights for proper fruit detection. When testing three different strawberry harvesting robots Kondo et al. (1998) mentioned that there was also a constant need for artificial light sources although the ripen fruits had red colour. Similar artificial lighting requirements were suggested by Tian et al. (1997), when sensing of different plants, and discrimination of weeds from crop plants and soil was studied by using visual system under controlled illumination. As noticed by Steward and Tian (1998) a direct sunlight caused substantial intensity differences within the images due to shadows and reflection from shiny leaf surfaces, thus Peterson et al. (1999) installed a fibre-reinforced drapery on the apple harvesting robot to block the influence of natural light conditions and increased the accuracy in detection of red coloured 'Empire' variety up to 95%. 6 MONITORING GROWTH AND DEVELOPMENT OF APPLE FRUITS 'MALUS DOMESTICA' BORKH. For determining the colour and shapes of weeds in the cereal fields Perez et al. (2000) used a variant of well-known Red/Green ratio measurement for predicting the number of seedlings and estimating the relative leaf surface of crops and weeds more accurately. With additional development of the shape detection algorithm Stajnko and Lakota (2001) reported the establishment of a close correlation between the number of fruits estimated on the images captured on the sunny side of apple trees and manually counted fruits, while detecting yellow and green-yellow 'Golden Delicious' fruit at the harvesting. However, to create a homogenous illumination across the whole image the application of the artificial light source (flash) was required on the sunny as well as shadow side of the tree row. In our research, the number of fruits was also determined prior the harvesting period, when the colours of the fruits did not differ substantially from the colour of leaves. To overcome the problem of insufficient colour gradient caused by lighting characteristics, the image arithmetic of Fig. 1. A sample image of 'Gala' apple tree captured on August 8th 2003; (a) original RGB image, (b) R image, (c) after filtering, (d) binary image, (e) objects after first template, (f) final results 7 MONITORING GROWTH AND DEVELOPMENT OF APPLE FRUITS 'MALUS DOMESTICA' BORKH. the basic colour planes followed by a two-template application and objects classifier evaluation was used. The main objective of this paper is to demonstrate and evaluate the applicability of the method for predicting the number and the diameter of the apple fruits needed for calculating the current and future yield in the fruit tree plantation. MATERIAL AND METHODS During the vegetative period May-August 2002 and 2003, ten apple trees (Malus domestica Borkh.) were examined in the Faculty's apple orchard (lat. 46o32' N, long. 15o33'5 E). Apple trees were planted in 1999 using the variety 'Gala' grafted on the M9 rootstock. The experiments were performed each time around noon, to ensure equal effect of the sunlight. The following developing stages of apple fruits were selected for capturing images in the crucial stages of the fruit's growth and development: 1. stage - after fruit drop June 6 2002 May 26 2003 2. stage - one month later June 22 2002 June 26 2003 3. stage - one month later July 11 2002 July 9 2003 4. stage - beginning of ripening August 14 2002 August 8 2003 5. stage - collecting fruit August 20 2002 August 28 2003 Each time one series of pictures were captured on the sunny side of trees and another series on the shadow side, both from the distance of 2.0 m at an angle of 90° degrees to the planting row. Concurrently, on each photographed tree all fruits were manually counted and the diameter of ten sample fruits was measured by applying a sliding calliper. Results were later compared with the image analysis calculations. A CCD OLYMPUS 3030 camera with the Flash setting automatic program was used for capturing images with the resolution of 1280x960 pixels, since the best possible image quality 2048x1536 was found in our earlier experiments to reduce the processing speed significantly. The analysis of the images was carried out with a personal computer (PC) with a 350 MHz processor and 256 MB random access memory (RAM). Fruit detection algorithm The apple fruit recognition algorithm was based on colour and shape detection, however a chosen apple variety developed its colour differently according to the growing stage, therefore an adjusted algorithm had to be developed and tested. The discussion presented here will concern a whole set of images and will be illustrated with images showing one sample tree of 'Gala', chosen to be representative of the method's result. As seen from the RGB image (Fig. la), fruits could not be detected accurately, because the RGB intensity varied greatly according to the images exposure to both sunny and shadow side of the tree. To overcome this problem, in the first step the data of a representative image from each series was divided into three basic planes R, G, B. According to the histogram analysis, the G image was selected for the first stage and the R image for all further stages (Fig. lb). In the second stage the image was filtered using a specified size of kernel (3x3 pixels) to remove the remaining noise. Additionally, by applying the low-pass filter 'connec-tivity-4' two pixels were considered as part of the same object if they were horizontally or vertically adjacent (Fig. lc). Once pixels from leaves and fruits were established, in the third stage the differentiation between them proceeded on the binary images (Fig. Id) by using of two templates. The first elliptical template was chosen to detect as much spherical object as possible, while the majority of square parts (mounting) and elongated objects (leaves, branches) were rejected (Fig. le). After that with the second template representing the whole apple fruit of each developing stage, apples were differentiated from other spherical objects. The Table 1. Classification parameters of selected objects Apple fruit Apple fruil Leaf Leaf (whole) (partly) (whole) (paitly) Major axis X Upi>;i lufi -Xfcutioni rishi Minor axis A hutlom left —Auppcr riyhl Area it major axis minor axis Perimeter JT ¦ yj2< majoraxis + itrinoi axis' t 16 Ares Compactness Perimeter (majocaxis- minoraxis) Elongation (majocaxis+ minoraxis) LTP majocaxis (length to perimeter) perimete 1.02-1.06 0.52-0.55 0.40-0.42 0.57-0.59 0.08-0.10 0.51-0.54 0.61-0.63 0.22-0.24 0.30-0.32 0.33-0.35 0.38-0.39 0.36-0 3 'selected limits depend on the developing stage result of a two stage processing is shown in Figure If. As seen, a misidentification between leaves and fruits might occur when capturing images under natural conditions due to the similar illumination of different objects. To overcome this problem the standard morphological characteristics were calculated for each detected object and evaluated according to selected parameters of the four typical classes (Table 1). Prior the automatic evaluation classification the system was trained with a given objects separated from a part of the captured images on the same day the evaluation took place. The first class represents the whole apple fruit, the second the part of an apple fruit and two others are the leaf and a part of the leaf, respectively. The applied training parameters are represented in the Table 1. As shown, the most suitable parameters for significant differences between apples and the noise classes were the compactness, elongation and LTP (length to perimeter). Only objects completing the all criteria of borders values for apple fruit 'whole' and apple fruit 'partly' were finally accepted as fruits in the result image (Fig. If). The data of objects was later used for calculating the diameter of the fruit according to the pixel/mm proportion and estimating the future yield as already explained by Stajnko et al. (2004). 8 MONITORING GROWTH AND DEVELOPMENT OF APPLE FRUITS 'MALUS DOMESTICA' BORKH. Estimation of future yield The number of fruits and the average diameter of detected fruits on each image were the basis for estimating the current yield. A file with recorded fruit characteristics as well the yield per image was stored for conducting a statistical analysis. For calculating current yield on the image, the following equations were applied: Table 3. Number of apple fruits in 2003 (N=10) N0,4059D2-9602 (1) 70° where Yt represents the yield per tree in kg, N the number of fruits per tree, D the average value of the longest segment. The equation is based on a transformed function derived by Weite (1990) and allows a direct calculation of the weight from the fruit's diameter. He showed that during the fruit growing, the relative increase in diameter was proportional to the relative fruit weight increase. In our research for the first four samplings the average fruit yield per tree was calculated on the basis of average fruit's diameter measured manually by the sliding calliper and compared to the estimated yield based image analysis. Contrary, at harvesting the coefficient was calculated between weighted and estimated yield. For performing the above described algorithms the IMAQ Vision 4.1.1. and Labview 5.O.I. from National Instruments® was used in our investigation. The statistical analyses of manually and by image analysis obtained results were performed using SPSS Package Program. RESULTS AND DISCUSSION May 26 June 26 July 9 A gust 8 Au gusl 28 M IA M IA M IA M IA M IA Min 24 29 23 22 23 22 23 16 14 14 Max 50 68 57 68 48 49 48 50 47 49 Mean 39.7 40.7 39.7 37.8 37.6 36.5 37.6 34.0 33.6 31.6 S.dev 11.0 !2.1 11.1 13.4 9.9 10.0 9.9 10.2 12.1 12.2 CV. 27.8 29.9 27.9 35.4 26.3 27.4 26.3 29.9 35.9 38.6 Corr.coef. 0.76 0.79 0.76 0.89 0.92* * The apple fruit diameters per tree The average fruits diameters per tree predicted for different developing stages of apple are shown in the Tables 4 and 5. It is clearly shown, that the average fruit diameter per tree was lower than the actual fruit diameters at all developing stages during the vegetation period 2002, while in 2003 it was practically the same. Contrary, to the number of fruits per tree, the correlation coefficient varied from r=0.33 to r=0.80. The reason for lower coefficients lies in the underestimation of the apple fruit's diameter detected by the fruit detection algorithm, which is actually based on the longest segment measurements. Namely, it was shown during the research, that the algorithm was accurate sufficiently, if a whole apple fruit was detected or a part of it was clearly seen. However, leaves, branches and other fruits sometimes hide the edges of fruits, thus lower fruit diameter was measured by image analysis and lower correlation coefficient was obtained. Therefore, it is suggested to develop a more advanced algorithm for calculating the apple diameter and to include a long-term measurement of each variety in the database. The number of fruits per tree The number of apple fruits per tree detected by the image analysis and manually counted is presented in the Table 2, 3. As stated earlier the fruit detection algorithm was tested for different abounded apple trees with 12 to 68 fruits. However, because of the dry and hot weather in the summer, the number of fruits slightly decreased in both years from June to end August. Contrary, correlation coefficient (r) is varying from 0.83 to 0.88, depending on different developing stages of fruits. The lowest correlation was obtained in first measurements, on June 22 2002 (r=0.70) and May 26 2003 (r=0.76), while the highest correlation r =0.92 and r= 0.91 was established in both years at harvesting. The reason for the increase of the correlation coefficient values during the maturity period was changes of the fruits' colour and the diameter. Similar observations were reported by Kondo et al. (1998) for robotic harvesting of strawberries and Kataoka et al. (1999) when testing the robot for apple harvesting. Table 4. The average diameters of apple fruits (in mm) in 2002 (N=10) Table 2. Number of appl ä fruits in 2002 (N= 10) May 23 June 22 July 11 August 14 August 20 M IA M IA M IA M IA M IA Min 14 12 14 16 14 18 14 19 12 14 Maj 42 46 42 40 42 42 42 43 42 43 Mean 26.8 31 26.8 28.1 26.8 29.3 26.8 29.8 24.5 27.9 S.dev. 9.9 12.1 9.9 8.4 9.9 10.1 9.5 9.8 9.7 10.9 CV. 37.1 39.0 37.1 29.8 37.1 3.7 35.5 32.8 39.7 38.9 'orr.coef. 0.79 0.70' 0.87" 0.88 0.93" M ...manually [A ...image analysis ' p<0,05 "p<0,00l Mav 23 June 22 July 1 Augus 14 August 20 M IA M 1A M IA M IA M IA Min 20 20 32 30 52 40 67 49 70 55 Max 59 52 71 40 59 100 79 116 81 85 Mean 39 39 51 49 56 57 70 69 78 76 S.dev 14.7 12.0 14.7 12.6 2.2 214 3.5 19.9 3.3 9.9 CV. 37.6 30.8 28.8 25.7 3.9 37.5 4.9 28.8 4,2 13.0 Corr.coef. 0.77 * 0.80 0.78 0.88 0.77 Table 5. The average diameters of apple fruits (in mm) in 2003 (N=10) May 26 July 9 August 8 August 28 M IA M IA M IA M IA M IA Min 37 35 50 49 46 48 71 69 64 65 Max 48 56 58 64 64 72 80 82 89 88 Mean 42 41 54 53 58 58 76 76 79 80 S.dev, 34 5.9 2.7 2.9 5.7 7.8 2.9 5.2 6.1 6.8 CV. S.l 14.3 5.0 5.6 9.9 13.4 3.8 6.8 7.8 8.5 Corr.coef. 0.35 0.33 0.75' 0.80 0.77 Estimation of fruit yield The current mass of fruits per tree was estimated by applying equation 1, whereas the number of fruits and the average fruit diameter from each developing stage were used for estimating the yield. As seen from Table 6 and 7, the average estimated yield per tree was increasing in both years from the end of fruit tinning in May till the harvest in August, however the yield per tree was slightly overestimat- 9 MONITORING GROWTH AND DEVELOPMENT OF APPLE FRUITS 'MALUS DOMESTICA' BORKH. Table 6. The apple yield per tree (kg) in 2002 (N=10) May 23 June 22 July 11 Avgus 14 Ai gust 20 M 1A M IA M IA M IA M 1A Min 0.12 0.14 0.21 0,28 0.85 0.44 1,53 0,98 3.16 1.83 Max 2.91 2,29 3,22 3.17 3,48 7.82 3.60 9.19 7,36 S.98 Mean 0.95 0.81 1.57 1.30 .84 2.31 2.35 2.71 3.14 4.27 S.cicv 0.S2 0.69 1.55 1.02 0.73 2.29 0.66 2.45 1.43 2.37 CV. S6.3 85.S 98.7 78.5 39.7 99.1 2S.I 90.4 45.6 55.5 L'orr.cocf 0.86 0.7S 0.87 0.88 ).5S Corr.coefi . 0.2: O.lf O.OS 0.09 3.58 Table 7. The apple yield per tree (kg) in 2003 (N=10J May 26 June 26 July 9 August ö August 28 M 1A M 1A M IA M IA M LA Min 0.65 0.57 1.47 0.92 1,98 1.23 3,72 2.83 4.20 3.34 Max 1.46 1.79 3,11 3.04 3,75 4.22 7,48 6,48 9,82 8,20 Mean 0.97 0.99 2.17 1.93 2.53 2.47 5.56 4.95 7.64 5.49 S.ckv. 0.26 D.41 0.50 0.64 0.73 0.S2 1.22 1.28 2.45 1.97 CV. 26.8 41.4 23.0 33.2 28.9 33.2 21.9 25.9 32.1 35.S Corr.coef 0.2S 0.64 1.S4 0.66 0.70 Corr.cocf.i.^ 0.06 0 69 0.25 0.50 0.70 ed in 2002 and underestimated in 2003, as it was the number of fruits on which the calculations based. Consequently, the correlation coefficient between manual measurements and image estimation also varied from the lowest in May 2003 (r=0.28) to the highest in July 2003 (r=0.84). When representing and comparing the manual measurements with the estimation of the average yield per tree 5 -4,5 4 "or S 3,5 a 3 a d 2,5- >- LU < c a. LÜ >1,5-1 0,5 0 GALA 2002 r=0.90 23.05. 22.06. 11.07. DATE 14.08, 20.08. -MANUAL MEASUREMENTS —IMAGE ANALYSIS Fig. 2. Growing curves of the yield development in 2002 estimated by manual measurements and image analysis graphically (Fig. 2, 3), almost identical growing curves were estimated in both years. Thus, it could be concluded that the developed algorithm may represent a good tool for the early determination of fruit development and yield estimation. CONCLUSION A new approach for counting apple fruits on trees and estimating the diameter and the current yield under artificial lighting (flash) fruit tree plantation conditions was analysed 7,5-7 -6,5-aT 6 Ol -5,5 t 5 3 4,5 _i a 4 ^ 3,5 i 3- u ? s- > L,3 < 2 1,5 1 -0,5 0 GALA 2003 r=0.98 26.05. 26.06. 09.07. 08.08. 13.08. DATE -•-MANUAL MEASUREMENTS — IMAGE ANALYSIS Fig. 3. Growing curves of the yield development in 2003 estimated by manual measurements and image analysis in our research. The investigated measuring technique based on RGB imaging and analysis procedure was used successfully during the whole growing period of fruits from May to August in 2002 and 2003 in all cases when only a small part of the apple fruit was separated from the background. The system enables faster sampling and evaluation of larger plantation than it is possible with the current manual method. However, it was not always able to distinguish between fruits and leaves growing deep in the tree-crown. Therefore, future work should be focused on improving the algorithm by implementing the shape recognition procedure to the algorithm, so that it would be possible to detect also partially hidden spherical objects by obtaining a set of pixels belonging to the boundaries of apples. REFERENCES 1. Grand D'Esnon A, Rabate! G, Pellenc R. Magali: a self-propelled robot to pick apples, ASAE paper No. 87-1037, 1987. 2. Jimenez AR, Jain AK, Ceres R, Pons JL. Automatic fruit recognition: a survey and new results using Range/Attenuation images, Pattern Recognition 1999;32:1719-36. 3. Juste F, Sevilla F. Citrus: A European project to study the robotic harvesting of Oranges, in: Proc. 3rd Int.Symp. Fruit, Nut and Vegetable Harvesting Mechanization, Denmark, Sweden, Norway, 1991;331-38. 4. Kataoka T, Bulanon DM, Hiroma T, Ota Y. Performance of Robotic Hand for Apple Harvesting, ASAE Paper No.993003, 1999. 5. Kondo NK, Hisaeda A, Monta M. Development of Strawberry Harvesting. Robotic Hand, ASAE Paper No.983117, 1998. 6. Lambrechts G. Apple EU 2001 Forecast 2001, Prognosfruit 2001, http://cmlag.fgov.be/dg2/fr/Communications/prognos-algemeen.pdf., 2001. 7. Perez AJ, Lopez F, Benlloch JV, Christensen S. Colour and shape analysis techniques for weed detection in cereal feels. Comput. Electron. Agric. 10 MONITORING GROWTH AND DEVELOPMENT OF APPLE FRUITS 'MALUS DOMESTICA' BORKH. 2000;25:197-212. 8. Peterson DL, Anger WC, Bennedsen BS, Wolford SD.. A system approach to robotic bulk harvesting of apples. ASAE Paper No. 99-1075, 1999. 9. Ramos K, Lieben SM. Prognosfniit 2003 - European Crop Forecast Convention, 7 s., http://www.fas.usda.gov/gainfiles/200308/l4598 5760.doc. 2003. 10. Stajnko D, Lakota M. Using image processing and analysis techniques for counting apple fruits in the orchard, Horticultural Science (Prague) 2001; 28(3): 95-99- 11. Stajnko D, Lakota M, Hočevar M. Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Comput. Electron. Agric. 2004;42:31-42, 12. Steward BL, Tian LF. Real-Time Machine Vision Weed-Sensing. ASAE Paper No. 98-3033, 1998. 13. Tian L, Slaughter DC, Norris RF. Outdoor field machine vision identifica- tion of tomato seedlings for automated weed control. Transactions of ASAE1997;40:(6):176l-68. 14. Weite HE Forecasting harvest fruit size during the growing season. Acta Horticulturae 1990;276:275-82. 15. Winter, F. Modelling the biological and economic development of an apple orchard. Acta Horticulturae 1986;160:353-60. Received: June 11, 2004 Accepted: November 10, 2004 11