ije(MlDEM
\ Inurnal of M
Informacije |
^ Journal of Microelectronics,
Electronic Components and Materials Vol. 42, No. 1 (2012), 11 - 17
Quality Control of Automotive Switches with help of Digital Camera
Grega Bizjak*, Matej Bernard Kobav, Blaž Luin Univerza v Ljubljani, Fakulteta za elektrotehniko
Abstract: For the purpose of quality control at one Slovenian automotive parts producer a system for luminance measurement based on digital camera was developed. The system consists of black chamber in which the measured specimen and the camera are placed. They are positioned very close to each other with the distance between them limited with the minimal focus distance of the camera which is about few millimeters. Such a short distance assures that in spite of rather small symbol on a car switch there is still enough information on the picture that the luminance of the symbol can be calculated.
Special software automatically takes the picture, transfers it into the computer and analyzes it according to the car manufacturer standard. First the boundaries of the symbol are located on the pictures and then the measurement areas are found. In the next step, the luminance of the measured area is calculated. On one symbol two or three measuring areas are defined. If the luminance of the measuring areas is in the allowed interval, the switch is adequate to be mounted in a car, if not, the switch must be discarded. The result of the analysis (adequate, not adequate) is clearly stated on the screen. As whole measuring process is automatic no special knowledge is needed to manage the measuring system.
Keywords: quality control, automotive industry, digital image processing, luminance measuring with digital camera
Kontrola kakovosti avtomobilskih stikal s pomočjo digitalne kamere
Izvleček: V proizvodnji sestavnih delov za avtomobilsko industrijo je ustrezna končna kontrola izdelkov danes obvezna. V okviru končne kontrole pa je potrebno izdelek zanesljivo in čim hitreje preveriti. Zato je bil za slovenskega izdelovalca stikal za vgradnjo v avtomobile izdelan sistem za končno kontrolo svetlosti znakov na stikalih. Sistem temelji na uporabi digitalne kamere in ustrezne programske opreme. V članku je najprej predstavljena metoda meritve svetlosti osvetljenih simbolov na avtomobilskih stikalih. Namesto klasičnega merilnika svetlosti in črne sobe je pri opisanem postopku uporabljena digitalna kamera ter posebna črna komora. V komori sta merjeno stikalo in digitalna kamera nameščena na ustrezno kratki razdalji, ki omogoča ostro zajemanje slike, ustrezno razpoznavanje detajlom na sliki in izračun svetlosti simbola na stikalu. Z digitalno kamero najprej zajamemo sliko osvetljenega simbola in jo prenesemo na računalnik. S posebno programsko opremo se slika nato obdela, poiščejo se meje osvetljenega simbola in izračunajo se svetlosti referenčnih točk. Na podlagi izračunanih svetlosti se poda ocena o skladnosti testiranega vzorca z proizvajalčevim standardom. Rezultat analize (ustrezno, neustrezno stikalo) se jasno izpiše na zaslonu računalnika. Na posameznem simbolu se lahko upošteva dve ali več referenčnih točk. Celoten postopek od zajema slike preko določitve meja simbola, izračuna svetlosti in ocene ustreznosti je popolnoma avtomatiziran. Tako lahko z napravo rokujejo tudi delavci brez posebnega znanja o fotometriji in merjenju svetlosti. Kljub temu da je metoda za razliko od klasične meritve svetlosti precej hitrejša in lažje izvedljiva še vedno zagotavlja ustrezno zanesljivost. Poleg tega pa so tudi stroški za potrebna oprema precej nižji kot pri klasični merilni opremi. Tako sama naprava kot tudi programska oprema sta bili razviti v laboratoriju za razsvetljavo in fotometrijo na ljubljanski Fakulteti za elektrotehniko.
Ključne besede: kontrola kakovosti, avtomobilska industrija, obdelava digitalnih slik, meritev svetlosti z digitalno kamero
' Corresponding Author's e-mail:grega.bizjak@fe.uni-lj.si
1. Introduction	products are different switches for the car interior for
a large European car manufacturer. Due to the strong
For the purpose of quality control at one Slovenian	demands of the company internal standardization the
automotive parts producer a system for luminance	parts producer needs to have efficient quality control
measurement based on digital camera was devel-	in order to preserve orders. To control the luminance
oped at Laboratory of Lighting and Photometry. The	of the symbol on switches, the Slovenian produces had
two possibilities. One was the use of a luminance meter in a dark room. This measuring procedure needs a lot of time and some special equipment like a dark room, photometric bench etc. are needed together with the luminance meter. It is also very difficult to measure luminance of very small surfaces, illuminated with almost monochromatic light, at a short distance. The second possibility was the use of a luminance measuring device, based on digital camera.
Figure 1: Developed device for measurement of luminance of symbols on car switches.
In close cooperation with the mentioned Slovenian automotive parts producer such a luminance measuring device was developed in the Laboratory of lighting and photometry. First, the prototype device was constructed based on the commercial digital camera with 5 million pixels and possibility to control it through a USB port. The device was tested in the production quality control and proved to be very useful although clumsy to use due to some construction details [3]. So the second device, described in this paper, was constructed.
2. Design of the Black chamber
To prevent the external light sources to disturb the measuring results the measured switch and the camera were placed in a black chamber. Apart from this the chamber, build as a small cabined, also provides a power supply for the camera and switch sample and a USB connection for the camera. Inside the container is painted black to prevent light reflections. To enable easy exchange of the measured specimens, both the specimen and the camera are placed on the drawers.
Figure 2: Measured specimen mounted on a stand in a drawer.
Figure 3: Digital camera is positioned on the top of the measured specimen.
The measured specimen of a car switch is mounted on a special stand in the drawer (Figure 2) and connected to the power supply. The digital camera is mounted in the second drawer on top of the measured specimen (Figure 3). The specimen and the camera are positioned and adjusted so, that the distance between the specimen and camera lens assures focused picture at the closest possible distance. The camera is connected to the power supply and to the computer with the USB cable. After specimen is mounted on the stand, the drawer is closed to provide a total darkness inside the chamber.
3. Choice of the camera
= {(x, y)}, i(x, y) - i(x- 1, y) > T, x = 0, x-> xm
(1)
There are many different digital cameras on the market. But to be suitable for our application, the camera should fulfil following criteria:
•	the light sensitivity should be adequate to luminance range of measured symbols;
•	the minimal focus distance should be short enough so the taken picture of symbol covers most of the canvas;
•	it must be possible to control the camera remotely from the computer via USB port.
First requirement is that the camera is able to sense luminance range we were about to measure. The luminance of the symbols on car switches is normally in range from 2 to 60 cd/m2. Nearly every consumer camera fulfils this requirement. Ability to focus to a short distance of approximately 20 mm as the switch is relatively small significantly narrowed our choice of suitable cameras. The last important requirement is that the camera can be completely remotely controlled. This means that it should be possible to remotely set zoom, focus distance, aperture width and shutter speed. To sum up image should be remotely captured and transferred to the computer. Most of the remote controllable cameras use protocol PTP (picture transfer protocol), which is standardized as ISO 15740. Its default network transport media is USB (universal serial bus).
The camera that fulfilled our requirements is a consumer model Canon G7. Apart from this one, there are many other suitable cameras on the market.
4. Segmentation of the captured image and calculation of the luminance
After the sample has been put into the chamber and turned on, the image is captured and transferred to the software on computer. To be able to calculate the luminance of the symbol, first the boundaries of the illuminated symbol have to be found. This is accomplished by detecting the outer edges on all sides of the symbol. Therefore the procedure depends on the shape of the sign. In the following example we present a procedure for a nearly rectangular shaped sign like on the switch for heating the back window. In this case we have to detect four outer edges.
A first point on a left vertical edge is found by searching from left to the right for the first point in a row at which an increase in light intensity is larger than a predefined threshold.
where i(x, y) denotes light intensity of a point with coordinates (x, y) and T represents a predefined threshold.
Subsequently points on right vertical edge, upper horizontal and lower horizontal edges are found:
V, = {(x, y)}, i(x, y) - i(x + 1, y) > T, x = xmax, x-> 0 (2)
H^ = {(x,y)}, i(x,y) - i(x,y- 1) > T , y = 0,y->and (3)
HI = {(x, y)}, i(x, y) - i(x, y + 1) > T , x = 0 , -> 0 . (4)
After we find the first point on one edge the direction of the edge is determined using simple linear regression model. For horizontal sets
y=Oh+ßHx+Sh ,	(5)
has been used and for vertical sets x = Ov + ßvy + Sv . (6)
The estimates of regression parameters for the equation (5) are as follows
. nSxY- nSx S^ ^ Sy - ßH S^
ßv=nS"---"^-----" aH=-
XX ^X
n
(7)
and for the equation (6) as follows
nSXY - nSx	^	Sy - ßv SX	^
ßv = --------------- Ov =--------------, where	(8)
nSyy - Sy Sy	n
SXX = xi2 + x22 + ■■■ + xn2
Syy=y2+y2 + ■■■ + y"
Sxy=xiy2+x2y2+■■■+x,yn
S„ = x, + x, + ■■■ + x
X 1 2	n
Sy=yi + y2+■■■ + yn
(9) (10) (11) (12) (13)
After edges have been determined, corner points T. (x, y) can be calculated by finding intersections of the edges. On this basis outer borders of the symbol on a switch are determined. The next step is positioning of the measuring points on the symbol. The measuring points are circular areas on the illuminated part of the symbol. They are placed according to the car producer standard in the corners or in the middle of the lines. The dimension (radius) of the measuring point is 80 % of the symbol line width.
Figure 4: An image of the illuminated switch with edges and measuring points detected automatically and marked.
When measuring points are known, it is necessary to calculate the average luminance of area, which is covered by the measuring point. For that purpose, first the sRGB values of each pixel in the measuring area need to be transformed to CIE Lab color space. This is done by following transformations [2].
•	first from sRGB to RGB color space,
•	second from RGB to CIE XYZ color space and
•	third from CIE XYZ to CIE Lab color space.
All of the sRGB values are converted into RGB values in the same manner. Let R, G and B denote sRGB color values and r, g and b denote RGB color values. Then the transformation among them is defined by the following equations:
r =
R
--------, R < 0.04045
12.92
R + 0.055 ^2-4 , 1.055
(14)
, R > 0.04045
g =
--------, G < 0.04045
12.92
G + 0.055 , 1.055 ,
(15)
, G > 0.04045
b =
-------- , B < 0.04045
12.92
B + 0.055 1.055
(16)
, B > 0.04045
With obtained RGB values, it is possible to calculate CIE XYZ values with reference white D65 from them using the following equation:
X	=	r		0.412 0.213	0.019	
Y		g		0.357 0.715	0.119	(17)
_Z _		b_		0.180 0.722	0.950	
From CIE XYZ color space we calculate lightness component of CIE Lab color space with:
L* =
' 116 3V7 - 16, Y > 0.008856 , 903.3 Y, Y < 0.008856
(18)
Finally the luminance of the pixel can be calculated. Relation between the pixel lightness L* and luminance L is defined by the following equation [1]:
L = A(EV)eB<EV>L' .
(19)
Therefore our luminance value depends on camera parameter exposition value EV, which is described by the formula:
EV = log2
N2 t
(20)
where N is relative aperture (f-number) and t is shutter speed.
5. Calibration of the device
As one can see from the equation 19, the luminance of the symbol is calculated from the lightness (obtained from the picture data) with help of constants A and B. In the process of calibration the constants A and B should be set so, that the calculated luminance of the measuring point is the same as the luminance of the same point measured with a reference luminance meter. To define constants A and B properly at least two measuring point with different luminance need to be used.
The constants are calculated by regressing average lightness L* of all pixels inside the measuring point with measured luminance L of the same point on the switch. In order to make calibration successful at least two calibration points with different luminance are required, but in practice we need more points for good accuracy. After constants A and B are properly adjusted, calibration is finished.
As a reference instrument the normal luminance meter class L is used. For the purpose of calibration first the luminance of measuring points on some selected specimens are measured in a dark room of a laboratory
2.4
2.4
with specimens and instrument mounted on an optical bench. Afterwards the same specimens are inserted into black chamber of the measuring device and the picture of the symbol is taken.
As mentioned before, the software itself has two functions: measurement and calibration. Use of the software part for the calibration was already described in previous chapter.
With the calibration part of the software program, the lightness values of the selected points can be obtained from the picture and used for the calculation of both constants. The calculation is normally done by the same part of the program, so the user just need to select the points and input the previously measured luminance of these points. The user should input at least two points, but he/she can also input more points to get better values for A and B. Beside this possibility both constants can also be calculated using some other tools or software and inputted in the program manually.
To obtain better measurement result with the device, the constants need to be set for each type of the switch (symbol) separately. The constants are saved in a software configuration data and need to be set just once although we recommend the user to repeat the calibration procedure after certain time interval to compensate possible changes in camera CCD sensor.
Figure 5: Part of the software for calibration of the measurement system.
After both constants are set, the software and so also the device is ready for use.
6. Software Program
From the user's point of view the software should be simple and intuitive to use. We kept in mind that users probably will be neither computer nor lighting experts. To accomplish this graphical user interface has been developed which allows users to be able to choose functions by clicking.
Figure 6: Main window of the software where type of measured switch is chosen.
Measurement is accomplished by selecting first the type of the sign we are about to measure with clicking on the right picture on software desktop. After clicking the "measurement" button the camera is initialized with appropriate parameters such as shutter speed, aperture and ISO sensitivity.
Figure 7: Settings which are send to camera before picture is taken.
The picture is then automatically captured, transferred to the computer and displayed in a window. In the next step the measuring points are automatically detected. To allow user to verify measuring points, they
are drawn on the displayed image. After calculation of luminance of measuring points the results are displayed in the lower part of the "Measurement preview window". On the left side of the results also a statement whether measured specimen complies with the standards or not is written together with green (complies) or red (not complies) symbol.
Figure 8: Measurement results, the specimen complies with the standard.
Figure 9: Measurement results of a specimen which does not comply with the standard.
Figure 10: Database with list of last performed measurements.
Figure 11: Results of a measurement stored in a database.
After measurement is done, the results can be saved in a database. On this point also a comment to be saved with the results can de added. Together with the results and comment also the picture and the calibration data are saved. Saved results are so documented for the quality control and can be later analyzed again if needed.
From the development point of view the software has been developed in c programming language under Linux 2.6 operating system because of a large choice of open source libraries. In order to remotely control the camera LIBPTP communication library has been used. Graphical user interface is implemented using GTK toolkit.
References
1.	COUTELIER, B., Luminance calibration of the Nikon Coolpix 990 digital camera, EPIC 2002 AIVC, October 2002, pp. 697-702
2.	ZAGAR, T., BIZJAK, G. Izracun svetlosti iz RGB vrednosti umerjenega digitalnega fotoaparata in razvoj aplikacije za analizo svetlosti, Faculty of electrical engineering, University of Ljubljana, Ljubljana 2004
3.	LUIN, B., KOBAV, M. B., BIZJAK, G. Luminance measurement device based on digital camera for quality control in automative industry. 26th Session of the CIE: 4.-11. July 2007, Beijing, China
Arrived: 13. 09. 2011 Accepted: 26. 1. 2012