Image Anal Stereol 2008;27:87-95 Original Research Paper BLIND CONTRAST ENHANCEMENT ASSESSMENT BY GRADIENT RATIOING AT VISIBLE EDGES Nicolas Hautière1, Jean-Philippe Tarel1, Didier Aubert2 and Éric Dumont1 1LCPC, Division for Road Operation, Signalling and Lighting, 58 bld Lefebvre, 75732 Paris Cedex 15, France; 2INRETS-LCPC, Vehicle-Infrastructure-Driver Interactions Research Unit, building 824, 14 route de la Minière, 78000 Versailles, France e-mail: nicolas.hautiere@lcpc.fr, jean-philippe.tarel@lcpc.fr, didier.aubert@inrets.fr, eric.dumont@lcpc.fr (Accepted May 7, 2008) ABSTRACT The contrast of outdoor images acquired under adverse weather conditions, especially foggy weather, is altered by the scattering of daylight by atmospheric particles. As a consequence, different methods have been designed to restore the contrast of these images. However, there is a lack of methodology to assess the performances of the methods or to rate them. Unlike image quality assessment or image restoration areas, there is no easy way to have a reference image, which makes the problem not straightforward to solve. In this paper, an approach is proposed which consists in computing the ratio between the gradient of the visible edges between the image before and after contrast restoration. In this way, an indicator of visibility enhancement is provided based on the concept of visibility level, commonly used in lighting engineering. Finally, the methodology is applied to contrast enhancement assessment and to the comparison of tone-mapping operators. Keywords: advanced driver assistance system, blind assessment, contrast restoration, contrast enhancement, edges segmentation, LIP model, tone-mapping, visibility level. INTRODUCTION The contrast of outdoor images acquired under adverse weather conditions, especially foggy weather, is altered by the scattering of daylight by atmospheric particles (Narasimhan and Nayar, 2002). As a consequence, different methods have been designed to restore their contrast, in order to maintain the performances of video-surveillance systems (Narasimhan and Nayar, 2003) or in-vehicle vision systems (Hautière et al., 2007) as much as possible. However, there is a lack of methodology to assess the performances of such methods, or to rate them. Since fog effects are volumetric, fog cannot be addressed like a classical image noise or degradation which might be added and then removed. Consequently, compared to image quality assessment (Sheikh et al., 2006) or image restoration (Guichard et al., 2002) areas, there is no easy way, synthetic images from 3D models put aside, to have a reference image, which makes the problem not straightforward to solve. In this article, a solution is proposed. First of all, visible edges in the image before and after contrast restoration are extracted. The rate of new visible edges is deduced. Then, the ratio of the gradient of the visible edges between both images is computed. Thanks to the concept of visibility level, proposed by Adrian (1989), it is shown that this coefficient corresponds to the visibility enhancement produced by the restoration algorithm. Finally, based on this result, an indicator of visibility enhancement is derived. The article is organized as follows. First, the visibility model of Adrian (1989) is presented as well as how to use it to derive a blind contrast restoration assessment method based on visible edges ratioing. Second, the proposed methodology is applied to assess the performances of a contrast restoration method of daytime fog images acquired using in-vehicle cameras. This method is summarized for completeness. Compared to classical contrast enhancement techniques, our approach is a restoration method, since it is based on a model of image degradation, which is typically done in remote sensing (Sadot et al., 1995). Finally, the proposed methodology is discussed and applied to two other topics: the assessment of histogram-based contrast enhancement techniques and the comparison of tone-mapping operators. MATERIALS AND METHODS VISIBILITY MODEL For non-periodic targets, visibility can be related to the (Weber) luminous contrast C, which is defined as: C DL Lb Lt—Lb Lb (1) _ 87 Hautière N et al: Blind contrast enhancement assessment where DL is the difference in luminance between target and background, Lt is the luminance of the target, Lb is the luminance of the background. The threshold luminance difference DLthreshold indicates a value at which a target becomes perceptible with a high probability. It depends among other things on target size and ambient light level, decreasing with increase of light level, but leveling off and hardly changing in the photopic domain. For suprathreshold contrasts, the visibility level (VL) of a target can be quantified by the coefficient: VL Cactual Cthreshold (2) At threshold, the visibility level equals one and above threshold it is greater than one. Combining Eq. 1 and Eq. 2, we have: VL \ / L b J actual \ / L b /threshold (3) As the background luminance Lb is the same for both conditions, then this equation reduces to: VL = DLactual/DLthreshold (4) In any given situation, it might be possible to measure the luminance of the target and its background, which gives DLactual. But to estimate VL, we also need to know the value of DLthreshold. This can be estimated using Adrian’s empirical target visibility model (Adrian, 1989). VISIBLE EDGES RATIOING The model which has been presented in the previous section can be used to predict the visibility of objects according to their size, their contrast, the lighting conditions, the age of the observer and the observation time. However, using complex images, i.e., an image which contains several objects on a non-uniform background, it is not straightforward to calculate the value of DLthreshold. Indeed, it is at least necessary to detect, segment and estimate the size of the different arbitrary objects present in the image, which still remains a challenging task in computer vision. To solely assess the performances of a contrast restoration method, it is not necessary to achieve such a complex process. Instead, following the approach described in (Hautière and Dumont, 2007), it is proposed to compute, for each pixel belonging to a visible edge in the restored image, the following coefficient r: r = f-1(DIr)/f-1(DIo) (5) where DIr denotes the gradient in the restored image, DIo the gradient in the original image and f the camera response function (Grossberg and Nayar, 2004). Then, if the camera response function is assumed to be linear, which is generally the case for CCD sensors, Eq. 5 becomes simply: r = DIr/DIo = DLr/DLo (6) Here, r is mathematically defined because only the gradients of visible edges in the restored image are considered. Hence, only pixels having a minimum contrast can be restored, which ensures that DIo is different from zero. Thereafter, assuming that an object in the image is composed of edges, Eq. 6 can thus be rewritten as: r = DLr/DL threshold DLo/DL threshold (7) where DLthreshold would be given by Adrian’s model. Finally, Eq. 7 becomes: r = VLr/VLo (8) where VLr denotes the visibility level of the considered object in the restored image and VLo the visibility level of the considered object in the original image. Consequently, the computation of r enables to compute the gain of visibility level produced by a contrast restoration method. The remaining difficulty is in detecting the visible edges in the images, and it depends on the type of images under consideration. In the following sections, this methodology is applied to images altered by daytime fog acquired using in-vehicle cameras. CONTRAST RESTORATION In this section, a contrast restoration method dedicated to in-vehicle applications is presented. First, a classical model of daytime fog visual effects is recalled. Then, a contrast restoration methodology is summarized and illustrated on different road scene configurations. Visual properties of fog The attenuation of luminance through the atmosphere was studied by Koschmieder (Middleton, 1952), who derived an equation relating the apparent luminance L of an object located at distance d to the “intrinsic” luminance L0 of this object, measured at close range: L =L0e -ßd + L{1-e-ßd) (9) 88 Image Anal Stereol 2008;27:87-95 Fig. 1. (a–c) Samples from three foggy image sequences acquired using an in-vehicle camera, named ‘Minière’, ‘Piste’ and ‘Vehicule’, respectively. (d–f) Same images with contrast restored using the proposed methodology. where L^ is the atmospheric luminance and ß is the extinction coefficient of fog. On the basis of this equation, Duntley developed a contrast attenuation law (Middleton, 1952), stating that a nearby object exhibiting contrast C0 with the background will be perceived at distance d with the following contrast: C=[(L-Loo)/Loo]e-ßd = C0e-ßd. (10) This expression serves to base the definition of a standard magnitude called “meteorological visibility distance” Vmet, i.e., the greatest distance at which a black object (C0 = - 1) with a suitable size can be seen in the sky on the horizon. With the threshold contrast set to 5% (CIE, 1987), this definition yields the following expression: Vmet 1 -— log(0.05) 3 (11) Restoration methodology Principle. In a foggy image, the intensity I of a pixel is the result of the camera response function f applied to Eq. 9. Assuming that f is linear, Eq. 9 becomes: I = f(L) = Re-bd +A¥(1-e-bd) (12) where R is the intrinsic intensity of the pixel, i.e., the intensity corresponding to the intrinsic luminance value of the corresponding scene point and A¥ is the background sky intensity. Hence, to restore the contrast, it is proposed to reverse Eq. 12, which becomes: R Iebd +A¥(1-ebd) (13) Assuming a flat world scene, it is possible to estimate (b, A¥) thanks to the existence of an inflection point on the representative curve of Eq. 12 (Lavenant et al., 2002; Hautière et al., 2006b). Therefore, in order to be able to correctly restore the scene contrast, the remaining problem is the estimation of the depth d of the pixels. Scene depth modeling. The depth distribution in a road scene can be roughly decomposedin three parts: the road surface, the sky and the surroundings. Such an heuristic model is proposed and is detailed in the following equations. The depth d of a pixel at coordinates (u,v) which does not belong to the sky region, i.e., whose intensity is lower than A¥ is given by: d = min(d1,d2) (14) where d1 models the depth of pixels belonging to the road surface, which is assumed to be a plane: _ 89 Hautière N et al: Blind contrast enhancement assessment d1 l if v > vh v - vh and d2 models the depth of verticals objects: d2 V k (u-uh)2 + (v-vh)2 (15) (16) In these equations, (uh,vh) denotes the vanishing point position in the image, l depends on the intrinsic and extrinsic camera parameters and k > l controls the relative importance of the vertical world with respect to the flat world. Finally, a clipping plane at = l- is used to limit the depth modeling errors c-vh d near the horizon line. A sample of such a scene model is given in Fig. 2. Fig. 2. A sample of scene depth model proposed for restoring the contrast combined with Eq. 13. One can see its three components: the road plane, the vertical surroundings and the sky region (in blue). This particular model was used to obtain Fig. 1d. (uh, vh) denotes the position of the vanishing point in the image. Algorithm. To correctly restore the contrast, according to the scene model given in the previous paragraph, the remaining task consists in finding the optimal values of k and c. One solution is to solve the following equation using Powell’s method: (k*,c*) = argmax Qfkc) + k-c c>0 (17) where Q is a norm of the local normalized correlation between the original image and the restored image. Indeed, the normalized correlation score between the original and the restored versions of a neighborhood should remain high. A decreasing normalized correlation means that the content of the original and restored neighborhoods differ. More details about this method as well as alternate algorithms are given in (Hautière et al., 2007). BLIND ASSESSMENT Originally, the local contrast estimator presented in this section has been developed to estimate the visibility distance using in-vehicle cameras (Hautière et al, 2006a). In this section, we show that it can also be used to assess the quality of a contrast restoration method. Visible edges segmentation Principle. In order to be consistent with the definition of the meteorological visibility distance proposed by (CIE, 1987), it is enough to consider the set of edges which have a local contrast above 5% so as to obtain the visible edges under daytime foggy weather. The LIP model (Jourlin and Pinoli, 2001) has introduced a definition of contrast well suited to digital images. In this definition, the contrast between two pixels x and y of an image f is given by: C(x,y) (f) = max[f(x), f(y) ] A min \f(x), f(y) ] , (18) where A denotes lip substraction. Naturally, this definition of contrast is consistent with the definition of contrast used in visual perception (Eq. 1). Then, the contrast associated to a border F which separates two adjacent regions follows: C F (f) where A addition. 1 card V and È\ denote + x yevCWf) (19) LIP multiplication and Implementation. To implement this definition of contrast between two adjacent regions, Köhler’s segmentation method has been used (Köhler, 1981). Let f be a gray level image. A couple of pixels (x,y) is said to be separated by the threshold s if two conditions are met. First, y G V4{x). Secondly, the condition (Eq. 20) is respected: min [f(x), f{y) ]y(s) is computed: Cx,y(s) = min \s-f(x)\ \s-f(y)\ max(s,f(x)) ' max(s,f(y)) (21) The mean contrast (Eq. 22) associated to F(s) is then computed: C(s) 1 å Cxy(s) cardF(s) , fF,. (22) _ _ _ _ 90 Image Anal Stereol 2008;27:87-95 8^b^9 ^ —3--------___ (b) X \ T^-;+^I.7 ' ' a - -~- -,^L_ ^^^*^^ ------------------------__--------____Jd) -------_ D ¦------¦---------------___ \W Fig. 3. Computation of local contrasts above 5%, assumed to be the visible edges by daytime fog, in the images of Fig. 1. The best threshold s0 verifies the following condition: s0 = argmaxC(s). (23) s?[0,255[ It is the threshold which delivers the best mean contrast along the associated border F(s0). Instead of using this method to binarize images, we use it to measure the contrast locally. The evaluated contrast equals 2C(s0) along the associated border F(s0). Finally, if 2C(s0) > 5%, F(s0) is considered to be a visible edge. Details about the implementation of this method can be found in (Hautière et al., 2006a). Descriptors no and nr denote respectively the cardinal numbers of the set of visible edges in the original image Io and in the contrast-restored image Ir. The latter set is denoted Ãr. First of all, we propose to compute e, the rate of new visible edges in Ir: e = nr — no no (24) The value of e evaluates the ability of the method to restore edges which were not visible in Io but are in Ir. In complement, we propose to compute r¯, the geometric mean of the ratios of VL defined by Eq. 8. The value of r¯ expresses the quality of the contrast restoration by the proposed method. Contrary to Eq. 24, this descriptor takes into account both invisible and visible edges in Io: r = exp nr Ptft (25) Finally, we propose to compute the number ns of pixels which are saturated (black or white) after applying the contrast restoration but were not before. We normalize this value by the size of the image, which gives the s indicator: (7 = ns dimx x d imy (26) where dimx and dimy denote respectively the width and the height of the image. RESULTS Samples from three foggy image sequences acquired using an in-vehicle camera, named ‘Minière’, ‘Piste’ and ‘Vehicule’, are given in Figs. 1a–c. The proposed contrast restoration methodology was applied to them. The outputs are given in Figs. 1d–f. Then, the local contrasts above 5%, assumed to be the visible edges in daytime fog, were extracted in the images of Fig. 1. The results are given in Fig. 3. Finally, the different proposed descriptors were computed to assess the contrast restoration. The value 91 Hautière N et al: Blind contrast enhancement assessment Fig. 4. False color map of r values computed on the pairs of images of Fig. 1. Each pixel shows the enhancement of visibility level induced by the contrast restoration algorithm here summarized. of r for each visible edge in Ir was computed and is shown in Fig. 4 using false colors. The main point to notice is that the visibility enhancement is higher for distant objects than for close objects, as expected. Then, descriptors (Eq. 24 and Eq. 25) have been computed for the images in Fig. 1 and for the results of a more classical histogram stretching algorithm. The results are given in Table 1. As expected, the proposed method performs better than the histogram stretching, which is a spatially invariant filter contrary to our method. However, the saturation s is not negligible especially in ‘Minière’ image. Table 1. Quantitative evaluation of two contrast restoration methods applied to the images in Fig. 1a,b,c using the descriptors (Eq. 24 and Eq. 25): (a) algorithm here summarized; (b) classical histogram stretching. (a) (b) s(%) s(%) Minière 1.4 2.6 2.2 Piste 1.1 1.8 0 Vehicule 1.6 1.7 0.1 0.25 1.1 0.1 0.33 1.3 0 0.4 1.1 0 DISCUSSION The proposed methodology allows to assess the performances of contrast restoration methods based on visual descriptors. However, it does not rate the fidelity of the contrast restoration method. It only measures how the visibility of objects in the scene is enhanced. Rating the fidelity can only be achieved with images of the scene with and without fog, which can be done using synthetic images. Notice that the proposed method is not able to assess the creation of visual artefacts. Furthermore, the proposed model is valid for 1 http://www.mathworks.com/ a linear model of camera. The contrast improvement of quasi-saturated pixels is thus not properly assessed. High dynamic range imaging sensors could help to solve this problem. Thereafter, it would be interesting to apply the proposed methodology to other types of contrast-degraded images. For instance, night-fog images acquired using in-vehicle cameras are very poorly contrasted. The dynamic range maximization described in (Jourlin and Pinoli, 2001) makes it possible to improve the visibility in such images. Köhler’s method can be used to detect visible edges in night images using a DCT transform and a Contrast Sensitivity Function (CSF) of the human visual system (Hautière and Aubert, 2006). It should thus be possible to apply the same methodology to assess the performances of the range maximization algorithm. OTHER POTENTIAL APPLICATIONS The proposed blind assessment method is not restricted to evaluate contrast restoration methods. We see at least two other applications which are illustrated now. The first is dealing with classical histogram based contrast enhancement algorithms. The second is dealing with the comparison of images obtained by different tone mapping operators. In both cases, the reference image is not known. Contrast enhancement The proposed methodology can be used to evaluate classical contrast enhancement techniques relying on histogram modifications. As an example, the effectiveness of the three functions proposed in the Matlab1 Image Processing Toolbox can be compared using the default settings: – “imadjust” increases the contrast of the image by mapping the values of the input intensity image to e e 92 Image Anal Stereol 2008;27:87-95 Fig. 5. (a) Original low contrast image; (b) contrast enhancement performed with “imadjust” Matlab function; (c) contrast enhancement performed with “histeq” Matlab function; (d) contrast enhancement performed with “adapthisteq” Matlab function; Second row: false color map of r values computed on the pairs of images. new values such that, by default, 1% of the data because it allows saturation and does not enhance is saturated at low and high intensities of the input contrast as well. data. – “histeq” performs histogram equalization. It enhances the contrast of images by transforming the intensity values so that the histogram of the output image approximately matches a specified histogram (uniform distribution by default). – “adapthisteq” performs contrast-limited adaptive histogram equalization. Unlike “histeq”, it operates on small data regions (tiles) rather than the entire image. The contrast of each tile is enhanced so that the histogram of each output region approximately matches the specified histogram (uniform distribution by default). The contrast enhancement can be limited in order to avoid amplifying the noise which might be present in the image. In each case, the aim is to increase the contrast without saturating and thus losing some visual information. Hence, good results are described by high values of e and r¯ and low values of s. These indicators are computed for one of the sample images used in Matlab and are given in Table 2. The third method gives the best result in terms of low saturation. However, the second method might be preferred, because it gives the best r¯ value. The first method gives the worst results Table 2. Contrast enhancement indicators computed on the images of Fig. 5. ers (%) “imadjust” 2.2 3.1 2.6 “histeq” 2.5 4.1 3.1 “adapthisteq” 2.8 3.2 0.0 Tone mapping The second potential application is quite different and deals with the tone-mapping of high dynamic range images. Tone-mapping consists in converting real-world luminances into displayable luminances. The resulting images have different visual aspects and it is difficult to judge which tone-mapping operator is the best. The proposed methodology allows to compare tone-mapped images of the same scene. However, it is not known in advance which image is best contrasted, so the images must be compared both ways. The images in the first row of Fig. 6 result from three different operators and are provided by the MaxPlanck-Institut für Informatik2. Fig. 6a is obtained through Ward’s operator (Ward Larson et al., 1997) which is known to maximize the contrast. Fig. 6b is 2http://www.mpi-inf.mpg.de/re source s/tmo/NewExperiment/TmoOverview.html 93 Hautière N et al: Blind contrast enhancement assessment (A) s (%) (A) — — — (B) 0.5 2.3 1.6 (C) 0.05 1.4 1.5 (B) e r s (%) -0.3 1.03 0.0 -0.3 1.02 0.0 (C) e r s (%) -0.04 1.06 0.1 0.4 1.7 0.14 Fig. 6. First row: Images tone-mapped through (A) Ward’s operator (Ward Larson et al., 1997), (B) Tumblin’s operator (Tumblin et al., 1999) and (C) Reinhard’s operator (Reinhard, 2002). Second row: corresponding false color map of r values for certain cases. (A) is compared in two ways with (B) and (C). Third row: comparison of the three operators based on the proposed indicators. obtained through Tumblin’s operator (Tumblin et al., 1999), whose aim is to preserve the luminosity of the scene. Fig. 6c is obtained by Reinhard’s operator (Reinhard, 2002) which aims to mimic photographic techniques. We denote these operators respectively A, B, and C. In the second row of Fig. 6, we compare (A) with respect to (B) and (C). (A) gives more contrasted results as expected (see Figs. 6A/B, A/C). However, it tends to saturate the light sources, which is shown in Figs. 6B/A, C/A. The proposed indicators are given in the table in the bottom of Fig. 6. CONCLUSION In this paper, the problem of the assessment of contrast restoration algorithms for weather-degraded images has been addressed. A solution based on visible edges ratioing has been proposed, which computes, for each pixel belonging to a visible object in the restored image, the visibility level (VL) enhancement produced by the algorithm. This method has been applied to daytime fog images acquired onboard a moving vehicle. In this context, the visible edges are assumed to be the pixels having a local contrast above 5%. An operator based on a segmentation algorithm has been proposed to extract such pixels and has been used to assess the performance of contrast restoration algorithms. A contrast restoration algorithm is summarized. It is based on a photometric model of fog and consists in inverting this model with an inferred depth distribution of the scene. Finally, we propose four descriptors of the enhancement: a map of VL enhancement for each pair of foggy and restored images, the geometric mean of VL enhancement, the rate of new visible edges and the percentage of saturated pixels produced by the restoration method. e 94 Image Anal Stereol 2008;27:87-95 The proposed methodology is generic and can find other applications such as the assessment of contrast enhancement techniques and the comparison of tone mapping algorithms. ACKNOWLEDGMENT The authors would like to thank Michel Jourlin, Université Jean Monnet, Saint-Etienne, France, for his advices and Roland Brémond, Laboratoire Central des Ponts et Chaussées, for his help on the tone-mapping. REFERENCES Adrian W (1989). Visibility of targets: Model for calculation. Light Res Tech 21:181–8. CIE (1987). International Lighting Vocabulary. No. 17.4. Grossberg M, Nayar SK (2004). Modelling the space of camera response functions. IEEE Trans Pattern Anal 26:1272–82. Guichard F, Moisan L, Morel JM (2002). A review of p.d.e. models in image processing and image analysis. J Phys IV 12:137–54. Hautière N, Aubert D (2006). Visible edges thresholding: a HVS based approach. In: Proc 18th Int Conf Pattern Recogn, 20–24August 2006, Hong-Kong,China, vol. 2, 155–8. Hautière N, Aubert D, Jourlin M (2006a). Measurement of local contrast in images, application to the measurement of visibility distance through use of an onboard camera. Trait Signal 23:145–58. Hautière N, Dumont E (2007). Assessment of visibility in complex road scenes using digital imaging. In: Proc 26th session CIE, 4–11 July 2007, Beijing, China, vol. 2, D4:96–9. Hautière N, Tarel JP, Aubert D (2007). Towards fog-free in-vehicle vision systems through contrast restoration. In: Proc IEEE Comput Vis Pattern Recog (CVPR 2007), 18–23 June 2007, Minneapolis, Minnesota, USA, 1–8. Hautière N, Tarel JP, Lavenant J, Aubert D (2006b). Automatic fog detection and estimation of visibility distance through use of an onboard camera. Mach Vision Appl 17:8–20. Jourlin M, Pinoli JC (2001). Logarithmic image processing. Adv Imag Elect Phys 115:129–94. Köhler R (1981). A segmentation system based on thresholding. Graph Model Im Proc 15:319–38. Lavenant J, Tarel JP, Aubert D (2002). Procédé de détermination de la distance de visibilité et procédé de détermination de la présence d’un brouillard. French patent 0201822 submitted by LCPC / INRETS. Middleton WEK (1952). Vision through the atmosphere. Toronto: University of Toronto Press. Narasimhan SG, Nayar SK (2002). Vision and the atmosphere. Int J Comput Vision 48:233–54. Narasimhan SG, Nayar SK (2003). Contrast restoration of weather degraded images. IEEE Trans Pattern Anal 25:713–24. Reinhard E (2002). Parameter estimation for photographic tone reproduction. J Graph Tools 7:45–52. Sadot D, Rosenfeld A, Shuker G, Kopeika NS (1995). High resolution restoration of images distorted by the atmosphere, based upon average atmospheric MTF. Opt Eng 4:1799–807. Sheikh H, Sabir M, Bovik A (2006). A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Trans Image Process 15:3440–51. Tumblin J, Hodgins JK, Guenter BK (1999). Two methods for display of high contrast images. ACM Trans Graphic 18:56–94. Ward Larson G, Rushmeier H, Piatko C (1997). A visibility matching tone reproduction operator for high dynamic range scenes. IEEE Trans Vis Comput Graphic 3:291– 306. 95