EXPLOITING 3D ULTRASOUND FOR FETAL DIAGNOSTIC PURPOSE THROUGH FACIAL LANDMARKING 


ENRICO VEZZETTI1, DOMENICO SPERANZA2, FEDERICA MARCOLIN1,3, GIULIA FRACASTORO1 AND GIORGIA BUSCICCHIO4 
1Dipartimento di Ingegneria Gestionale, Politecnico di Torino, Italy; 2Dipartimento di Ingegneria Civile e Meccanica, Universita degli Studi di Cassino e del Lazio Meridionale, Italy; 3Dipartimento di Ingegneria Gestionale, Politecnico di Torino, Italy; 4Department of Gynaecology and Obstetrics, Hospital of Senigallia-Marche, Italy. 
e-mails: enrico.vezzetti@polito.it; d.speranza@unicas.it; federica.marcolin@polito.it; giulia.fracastoro@studenti.polito.it; g.buscicchio@hotmail.it 
(Received December 9, 2013, accepted January 27, 2014) 
ABSTRACT 
In the last decade, three-dimensional landmarking has gained attention for different applications, such as face recognition for both identification of suspects and authentication, facial expression recognition, corrective and aesthetic surgery, syndrome study and diagnosis. This work focuses on the last one by proposing a geometrically-based landmark extraction algorithm aimed at diagnosing syndromes on babies before their birth. Pivotal role in this activity is the support provided by physicians and 3D ultrasound tools for working on real faces. In particular, the landmarking algorithm here proposed only relies on descriptors coming from Differential Geometry (Gaussian, mean, and principal curvatures, derivatives, coefficients of first and second fundamental forms, Shape and Curvedness indexes) and is tested on nine facial point clouds referred to nine babies taken by a three-dimensional ultrasound tool at different weeks' gestation. The results obtained, validated with the support of four practitioners, show that the localization is quite accurate. All errors lie in the range between 0 and 3.5 mm and the mean distance for each shell is in the range between 0.6 and 1.6 mm. The landmarks showing the highest errors are the ones belonging to the mouth region. Instead, the most precise landmark is the pronasal, on the nose tip, with a mean distance of 0.55 mm. Relying on current literature, this study is something missing in the state-of-the-art of the field, as present facial studies on 3D ultrasound do not work on automatic landmarking yet. 
Keywords: 3D echography, 3D face, 3D ultrasound, dysmorphisms, landmarking, syndrome diagnosis 

INTRODUCTION 
Fetal diagnosis has recently viewed a growth in the medical field. The earliest and earliest possibility to see if the baby "in mommy's tummy" is sane and grows well offers both to future mothers and practi-tioners the opportunity to state, months before the childbirth, if the baby needs or will need special care. Many syndromes involve physical dysmorphisms that, if seen early, could bring the doctors to follow the mother with a tailored medical path. These kinds of syndromes may affect the functioning of internal organs or of the whole organism, bring cognitive and behavioural deficits, or growth retardation, and cause peculiar physical and facial features. Especially in the earliest months of pregnancy, these defects could not be immediately identified, both because the baby's lineaments are not formed yet and because of poor accuracy of echography tools. 
To improve the quality of echographies, in the last decade 3D tools have been employed, although these have not "taken the field" yet. In this context, face or body landmarks were taken into consideration for prenatal diagnosis. Johnson et al. (2000) demon-strated that three-dimensional ultrasonography (US) could be used in identifying and localizing cleft lip, as well as normal facial anatomy, prenatally. The shown advantage of 3D US is that the rendered image provides landmarks for planar images. Rotten et al. worked on 3D sonography for diagnosing retrognathia and micro-gnathia (Rotten et al., 2002). Then, they identified some landmarks to be analyzed for studying facial features: forehead, nasal soft tissues and nasal bones, upper lip, hard palate, oral cavity, inferior lip, and chin. The objective was also to assess the use of 3D ultrasound imaging in fetal facial examination (Rotten and Levaillant, 2004). Pilu et al. (2006) dealt with detecting anomalies in midline structures of fetal brain using three-dimensional ultrasound. The diagnosis is made when the main cerebellum landmarks, the fas-tigium of the fourth ventricle and the main fissures, could not be identified. Paladini et al. (2006) used 3D ultrasound to show how abnormalities of the cerebellar vermis and posterior fossa could be differentiated sonographically thanks to the employment of a set of morphometric parameters. They focused on simple anatomical landmarks easily detectable and measurable on the midsagittal ultrasound image of the fetal head (tentorium, maximum craniocaudal vermian diameter, clivus). Rochelson et al. (2006) used 3D sonographic multiplanar display with the use of spatial rotation to identify landmarks in the coronal (midline between the orbits at the nasal bridge) and transverse (midline at the level of the anterior orbit) planes, for establishing a reproducible and consistent facial profile in the midsagittal plane. The final goal is to apply geometric morphometric statistical analysis to quantify shape differences in normal and abnormal fetal skulls. Faure et al. (2007) described a new three-dimensional ultra-sound rendering technique to examine the normal fetal posterior palate, providing diagnostic information. The sonographic visualization rates of seven defined anatomical landmarks of the fetal palate were computed for each gestational age: maxilla with alveolar ridge and tooth buds, maxilla palatine process, interpalata suture, palate bone horizontal plate, transverse palatal suture, posterior nasal spine and pterygoid process. Vinals et al. (2007) used 3D multiplanar and Volume Contrast Imaging (VCI) in the coronal plane to assess the integrity of the median fetal cerebral structures. Landmarks of structures such as the cerebellar vermis and the fourth ventricle were accurately defined through the use of the VCI filter. Bromley et al. (2007) examined whether the third-trimester fetus could be assessed sonographically using three-dimensional volume data sets, with the objective of determining fetal presentation, amniotic fluid volume, placental location, and estimating fetal weight. This identification was done through gross anatomical landmarks. Using as well 3D volumes, but of fetal face, Plasencia et al. (2007) investigated the effect of deviations from the exact midsagittal view on measurements of the fronto-maxillary facial angle at 11+0 to 13+6 weeks' gestation. Landmarks identifying the midsagittal plane of fetal face have been manually detected by sonographers. Pugash et al. (2008) discussed the relative value of both prenatal US and fetal Magnetic Resonance Imaging (MRI) for prenatal diagnosis, with the support of landmarks to identify face and body parts. Turan et al. (2009) used 12 heart landmarks for second-trimester examination of fetal heart using three-dimensional US techniques. Final aim was to accurately discriminate between normal and abnormal cardiac anatomy. Individual anatomic landmarks were identified in 89.7–99.1%. Wong et al. (2009) examined the secondary palate at various gestational ages by the aid of oblique planes from stored 3D ultrasound volumes of the fetal face. The uvula served as a reliable landmark for viewing the soft palate. 
The research group of Sepulveda, Martinez-Ten, and Wong also used three-dimensional volumes and 3D ultrasound to prenatally diagnose physical mal-formations. Firstly, they introduced the retronasal tri-angle as a new first-trimester sonographic landmark, useful in the early screening for cleft palate (Sepulveda et al., 2010) and for the presence or absence of the nasal bones (Martinez-Ten et al., 2010). Then, they dealt with micrognathia diagnosis in the first trimester of pregnancy. Facial landmarks were here cited as identifiers for various face parts (Sepulveda et al., 2012a). Finally, they determined whether systematic examination of primary and secondary palates aided in the identification of orofacial clefts in the first trimester (Martinez-Ten et al., 2012). They also revie-wed techniques, advantages, limitations, and clinical applications of 3D ultrasound and fetal MRI (Sepulveda et al., 2012b) and pointed out that anatomical land-marks had a key role in assessing fetus's normal structure (Sepulveda et al., 2012c). 
Another work group, the one of Lituania and Tonni, dealt with prenatal diagnosis. Firstly, they reported on the application of OmniView, a new 3D sonographic software, and its applications in the prenatal sono-graphic study of the fetal hard and soft palate, recog-nizable by the two main anatomic landmarks, the uvula and velum. As positive outcome, OmniView allows visualization of all anatomic landmarks of this specific targeted area: labia, primary palate, alveolar ridge, posterior palate, uvula, velum, and tongue (Tonni and Lituania, 2012). As a sequel, the soft palate and especially the uvula were used to diagnose clinical features of Stickler syndrome, a rare connective tissue disorder (Lituania and Tonni, 2013). 
Persico et al. (2010) used 3D ultrasound to investigate the effect of deviations from the exact midsagittal plane on the measurement of nasal-bone length at 16–24 weeks' gestation. For each parasagittal and oblique view obtained, they looked for the pre-sence or absence of all the sonographic landmarks used for identification of the midsagittal plane. The sonographic landmarks commonly used to examine the fetal profile and to measure nasal-bone length, including the nose, upper and lower lips, the maxilla and the chin, were also visible in parasagittal and oblique sections of the profile. Borrell et al. (2011) assessed fetal anatomy using three-dimensional volu-mes acquired at 11 to 13 weeks in 223 pregnancies to identify the appropriate sections for evaluation of ten fetal anatomy landmarks. A limitation of the study is that these body landmarks retrieved from the 3D volumes were not assessed prospectively on ultrasound, and thus comparison was not possible. Manganaro et al. (2011) evaluated cleft lip and palate through MRI and US, although not 3D. The main facial landmarks were measured and analyzed for each fetus (forehead, occiput, orbits, nose, lips, chin, mandible). 
As can be seen from the previous literature, in many cases landmarks are manually extracted or even represent a marginal or a landing point of the method. In other words, "landmarking" is not undertaken in this field yet. As a reply, this work stands for a preliminary study on facial soft-tissue landmark positioning on fetal 3D face shells, i.e., point clouds representing faces, extracted from three-dimensional echographies. Landmark extraction is a common pre-starting point for many applications involving faces. We focused on this phase to provide a tool for the subsequent steps of diagnosis. In particular, we have developed an algorithm for automatically extracting soft-tissue landmarks on babies' faces. The method employed to build up the algorithm totally relies on geometrical criteria. More in detail, descriptors taken from the Differential Geometry, such as mean and Gaussian curvatures, coefficients of the first and second fundamental forms, principal curvatures, and other geometrical descriptors such as the Shape and Curvedness Indexes, are used to identify landmark positions on faces. 
The reason for this research is that, although echographies are going towards 3D, procedures and techniques for analysing the achieved three-dimensional information and supporting the diagnosis are not in-parallel developed. As a matter of fact, 3D tools are mainly aimed at providing lifelike three-dimensional images of the babies, without concretely exploiting the real potentialities that a 3D tool could offer for supporting fetal diagnosis. 
METHODS 
A facial landmark is a point which all faces share and has a particular biological meaning. In this study we have considered soft-tissue landmarks, which lie on the skin and can be identified once the face of the subject is digitally reconstructed, using proper reverse 3D techniques. In human face up to fifty-nine soft-tissue landmarks could be collected, but in our study we have used only 13 landmarks, as shown in Fig. 1. We have chosen to work only with landmarks that lie in the central part of the face because the borders of the face are too much affected by noise and moreover it is difficult to have an echography of the whole face, with also the borders.  

 
Fig. 1. Landmarks detected by our proposed method: PN-pronasal, SN-subnasal, ALA-alae, EN-endocan-thions, N-nasion, IE-inner eyebrow, CH-chelion, LS-labrum superior, LI-labrum inferior. These landmarks form a pivotal set of core points for describing facial shapes and features. 
Considering the morphological features of the face, in order to extract the landmarks it is necessary to employ a refining procedure that firstly identifies the region, collecting the significant points, then extracts the specific landmarks. Relying on different peculiarities of different facial regions where the landmarks are located, different combinations of the first, second and mixed derivatives, the Coefficients of the Fundamental Forms E, F, G, e, f and g, the curvatures K, H, k1 and k2, and Shape and Curvedness Indexes S and C as descriptors have been employed. A short description of their meanings is reported in the Appendix. 
The localization and extraction processes used in the algorithm to detect the landmarks are explained and graphically represented. 
PRONASAL 
The pronasal (PN) is the point on the nose tip. It is surely the point most easily identifiable by human eye, especially because it is the most salient, when the face is well oriented. These are its most noticeable geometrical features: 
1.
 it has high values of the Shape Index (S > 0.55); 


2.
 in our reference is an absolute maximum, so it is a critical point (its first derivatives with respect to x and y are approximately equal to zero); 


3.
 the principal curvature k2 has a local (absolute, most of the times) maximum in it; 


4.
 the Gaussian curvature K has a local maximum in it. 

Actually our algorithm indentifies an area of interest through conditions 1 and 2 (in particular, the condition that the first derivative with respect to x is approximately equal to zero), then it extracts the landmark using condition 3, namely maximizing k2. The steps of the process are explained in the scheme below. 



 
SUBNASAL 
The subnasal (SN) is the point which lies exactly below the nose, in that little dimple above the mouth. These are its geometrical features: 
1.
 it is a critical point (its first derivatives with respect to x and y are approximately equal to zero); 


2.
 the coefficient f is close to zero and f(x,y+.y) < 0, f(x,y-.y) < 0; 


3.
 the coefficient e has a local maximum in it; 


4.
 the coefficient g has a local minimum in it. 

In order to extract this landmark, the algorithm first identifies a region in the neighborhood of the pronasal, then it uses condition 2 to narrow the area, then it extracts the landmark with condition 4, namely minimizing g. The steps of the process are explained in the scheme below. 



 
ALAE 
The alae (AL) are the two points which lie on the left and the right of the widest part of the nose. Their geometrical features are: 
1.
 they belong to the points whose Shape Index lies in the range corresponding to the surface of ridge (S Î [0.325,0.625]); 


2.
 the derivative of z with respect of Dx is positive on the right ala (from an external point of view) and negative on the left; 


3.
 the coefficient e is positive; 


4.
 the coefficient E has two local maximums in them. 

The elaborated algorithm identifies two areas of interest using conditions 1 and 2, then it extracts the two landmarks using condition 4. The steps of the process are explained in the scheme below. 



 
ENDOCANTHION 
The endocanthions (EN) are the two points at which the inner ends of the upper and lower eyelid meet. These are their geometrical features: 
1.
 they belong to the points whose Shape Index lies in the range corresponding to the surface of cup or rut; 


2.
 in our reference system, they are local minimum, so they are critical points; 


3.
 the second derivative of z with respect to x Dxx is negative in them; 


4.
 the second derivative of z with respect to y Dyy is negative in them; 


5.
 the second derivative of z Dxy is nearly 0 in them (DxyÎ [-0.1,0.1]); 


6.
 the coefficient F is negative on the right endo-canthion and positive on the left; 


7.
 the coefficient f is negative on the right endocanthion and positive on the left; 


8.
 the coefficient e has a local minimum in them. 

Our algorithm identifies two areas of interest using conditions 1, 3, 5, 6, then it extracts the landmarks with condition 8. The steps of the process are ex-plained in the scheme below. 



 
NASION 
The nasion (N) is a point at the top of the nose, nearly between the eyes. In the horizontal direction, it lies on the high part of the nose bone; in the vertical direction it is in the hollow under the forehead. These are its geometrical features: 
1.
 it belongs to the points whose Shape Index lies in the range corresponding to the surface of ridge, saddle ridge, saddle point or saddle rut; 


2.
 it is a critical point (its first derivatives with respect to x and y are approximately equal to zero); 


3.
 the mean curvature H is close to 0 (H Î (-0.5, 0.5)); 


4.
 the coefficient f is nearly 0 (f Î (-0.1,0.1)); 


5.
 the coefficient g has a local minimum; 


6.
 the Gaussian curvature K is negative in it; 


7.
 the principal curvature k1 has a local maximum. 

In order to localize the landmark, the algorithm starts selecting a region of interest starting from the two endocanthions and the pronasal. Then it uses conditions 1 and 2 to narrow the area and finally it uses condition 7 to extract the landmark. The steps of the process are explained in the scheme below. 



 
INNER EYEBROWS 
The inner eyebrows (IE) are that points which lie at the connection between the nose bone and the eye-brows themselves. These are their geometrical features: 
1.
 it belongs to the points whose Shape Index lies in the range corresponding to the surface of ridge or dome (S > 0.5); 


2.
 the coefficient g is strictly negative in them; 


3.
 the coefficient f is a local maximum in right IE and a local minimum in the left IE. 

The elaborated algorithm firstly indentifies two regions of interest starting from the endocanthions, then it narrows the areas using condition 1; finally it extracts the landmarks using condition 3. The steps of the process are explained in the scheme below. 



 
LABRUM SUPERIOR 
The labrum superior (LS) is the point which lies on the upper lip of the mouth approximately in the middle, on that little bump under the subnasal. The labrum superior is the most evident landmark of the mouth, so it is the first localized. These are its geo-metrical features: 
1.
 it belongs to the points whose Shape Index lies in the range corresponding to the surface of the dome; 


2.
 the coefficient g has a local minimum in it; 


3.
 the curvedness index C has a local maximum in it; 


4.
 the Gaussian curvature K has a local maximum in it. 

In order to extract the landmark, the process selects a region of interest starting from the subnasal, then it uses condition 1 to narrow the area and finally, with condition 3, it extracts the landmarks. The steps of the process are explained in the scheme below. 



 
LABRUM INFERIOR 
The labrum inferior (LI) is the point which lies on the lower lip of the mouth, approximately in the middle, closer the hollow located above the chin. Its geometrical features are the same of the labrum superior: 
1.
 it belongs to the points whose Shape Index lies in the range corresponding to the surface of the dome; 


2.
 the curvedness index C has a local maximum in it; 


3.
 the coefficient g has a local minimum in it; 


4.
 the Gaussian curvature K has a local maximum in it. 

In order to localize the LI, the algorithm identifies a region of interest using the coordinates of the labrum superior and of the subnasal. Then, the pro-cess to extract the landmark is equal to the one of the labrum superior. The steps of the process are ex-plained in the scheme below. 



 
CHELIONS 
The cheilions are the two points on the outer corners of the mouth, where the outer ends of the upper and lower lips meet. The geometrical features that we have indentified are: 
1.
 it belongs to the points whose Shape Index lies in the range corresponding to the surface of rut or cup; 


2.
 the coefficient f is close to 0 in them (f Î (-0.1, 0.1)); 


3.
 the absolute value of the coefficient e is maxi-mum in them; 


4.
 the second derivative Dxy has a local maximum in the right chelion and a local minimum in the left one. 

The process uses the coordinates of the labrum superior and inferior and of the two alae to identify a region of interest. Then it uses condition 1 to narrow the area and it extracts the landmark using condition 4.  



 
EXPERIMENTAL VALIDATION 
From September 2012 to January 2013, 30 three-dimensional volumes of 30 fetuses at 22–32 weeks' gestation were acquired. Written informed consent was obtained from the parents for publication of clinical details, clinical images, and videos. Principles outlined in the Declaration of Helsinki have been followed. 
Among these acquisitions, 9 were selected and processed for the purposes of the study. The remaining ones were excluded because of high noise or practical "inconveniences" such as baby's hands on face, or simply too much inaccuracy in the scanning process. 
The ultrasound equipment was a Voluson system (GE Healthcare, Wauwatosa, WI, USA), with a RAB 4–8 (real time 4D convex transducer probe). The GE RAB 4–8 has a frequency range of 4 to 8 MHz and is used for OB applications (Footprint 63.6×37.8 mm, FOV 70°, V 85°×70°). In Table 1 the used scan settings for each 3D static acquisition and the fetal age of each baby are shown.  
Using 4D VIEW software, it is possible to see the images acquired with the Voluson System on three orthogonal planes, i.e., axial, sagittal, and coronal (Fig. 2). The plane chosen for the facial shell modeling is the midsagittal (Fig. 3). 
The distance between two successive slices is 0.4 mm. For each slice composing the whole volume, the relative DICOM format file is created and stored. The final store is exported into Simpleware ScanIP software for the 3D model reconstruction.  

 
Table 1. Weeks' gestation and scan settings for each baby. 
Ultrasound examination 
 
Volume Ultrasound: GE Voluson e - Transducer: RAB 4–8-RS/OB 
 
Fetus 
 Scan Setting 
 
Name 
 Week 
 MI 
 Fr 
 Tls
 Quality
 Th
 B(°)
 V(°) 
 Mix 
 SRI 
 
Bart 
 32 
 0.9 
 3.7/11.0 cm/52 Hz
 0.1
 max 
 29
 50 
 65 
 50/50 
 II 
 
Lisa 
 32 
 1.1 
 4.1/9.7 cm/54 Hz 
 0.1
 max 
 30
 53 
 65 
 40/60 
 II 
 
Gio 
 22 
 1.1 
 3.6/10.2 cm/58 Hz
 0.1
 max 
 30
 47 
 65 
 40/60 
 II 
 
Gian 
 22 
 1.0 
 3.6/8.1 cm/68 Hz 
 0.1
 high 2 
 30
 46 
 65 
 40/60 
 II 
 
Paul 
 32 
 1.1 
 2.8/10.9 cm/49 Hz
 0.2
 max 
 30
 54 
 65 
 40/60 
 II 
 
Pie 
 32 
 1.1 
 1.7/8.1 cm/55 Hz 
 0.1
 max 
 30
 58 
 65 
 40/60 
 II 
 
Elena 
 32 
 0.9 
 4.2/10.8 cm/47 Hz
 0.1
 max 
 30
 56 
 65 
 40/60 
 II 
 
Fede 
 32 
 0.9 
 4.4/11.1 cm/51 Hz
 0.1
 max 
 30
 50 
 65 
 40/60 
 II 
 
Simon 
 32 
 1.1 
 3.5/9.6 cm/56 Hz 
 0.1
 max 
 30
 50 
 65 
 40/60 
 II 
 

2
 
Fig. 2. Multiplanar image. 
3
 
Fig. 3. Midsagittal plane. 
 

Face data were collected in points clouds, then these shells were imported in Matlab® and triangu-lated. The triangular mesh was then converted into a square grid, onto which a Matlab® algorithm for facial landmarking based on the theoretical foundations   
of the previous section was elaborated, implemented, and run. 
In the following paragraphs the results of the land-mark extraction procedures will be presented using one face as sample. 
PRONASAL 
The area of extraction of the pronasal is firstly reduced by selecting all the points whose Shape Index is greater than the threshold value 0.5 (Fig. 4).   
Then, among these points, the points with DxÎ [-u, u] were selected, where u is the instrument uncertainty (Fig. 5). Finally, the algorithm extracts the landmark maximizing the k2 (Fig. 6).  

pronasale1.bmp
 
Fig. 4. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red or yellow) by choosing only the points whose Shape Index is greater than 0.5. 
pronasale2.bmp
 
Fig. 5. On the left. Graphical representation of the derivative of z with respect to x. On the right. The narrowing of the area of interest (in red and yellow) by choosing, among the points selected in the previous step, only the ones whose derivative of z with respect to x is in the range [-u, u], where u is the instrument uncertainty. 
pronasale3.bmp
 
Fig. 6. On the left. Graphical representation of the principal curvature k2. On the right. The extraction of the landmark maximizing k2. 

SUBNASAL 
The first step of the process of extraction of the subnasal is defining a region of interest using the   
pronasal already extracted (Fig. 7). Then, this region is reduced using the conditions on f (Fig. 8). At the end, the landmark is extracted minimizing g (Fig. 9). 

sn_zona.bmp
 
Fig. 7. Region of interest for the extraction of the subnasal. 
subnasale1.bmp
 
Fig. 8. On the left. Graphical representation of the coefficient f. On the right. The narrowing of the area of interest by choosing only the points that satisfy the conditions on f. 
subnasale2.bmp
 
Fig. 9. On the left. Graphical representation of the coefficient g. On the right. The extraction of the landmark minimizing g. 
 

ALAE 
The two alae were localized using different geo-metrical descriptors: two conditions are firstly applied using the index S and the first derivative Dx, then the   
landmarks are extracted maximizing E. The condition on the derivative is only used to distinguish between the left and right part of the face, so the algorithm can distinguish between right ala and left ala. In Figs. 10–12 are shown the different steps of the process.  

alare1.bmp
 
Fig. 10. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red or yellow) by choosing only the points whose Shape Index lies in the range corresponding to the surface of rut. 
alare2.bmp
 
Fig. 11. On the left. Graphical representation of the derivative of z with respect to x. On the right. The narrowing of the area of interest (in red and yellow) by choosing, among the points selected in the previous step, only the ones whose derivative of z with respect to x is positive on the right side of the face and negative on the left side. 
alare3.bmp
 
Fig. 12. On the left. Graphical representation of the coefficient E. On the right. The extraction of the landmarks maximizing E. 

ENDOCANTHION 
The localization of endocathions was performed firstly selecting the points whose Shape Index lies in the range corresponding to the surfaces of cup and rut (Fig. 13), then the algorithm uses the conditions on the second derivatives (Figs. 14 and 15) and on the coefficient F (Fig. 16) to narrow the areas, as previously explained. At the end, the landmarks are extracted maximizing the coefficient e (Fig. 17). The condition on the coefficient F is only used to dis-tinguish between right and left endocanthion.  

endo1.bmp
 
Fig. 13. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red) by choosing only the points whose Shape Index lies in the range corresponding to the surface of cup. 
endo2.bmp
 
Fig. 14. On the left. Graphical representation of the second derivative of z with respect to x. On the right. The narrowing of the area of interest (in red) by choosing, among the points selected in the previous step, only the ones whose second derivative of z with respect to x is positive. 
endo3.bmp
 
Fig. 15. On the left. Graphical representation of the second derivative Dxy. On the right. The narrowing of the area of interest (in red) by choosing, among the points selected in the previous step, only the ones that satisfy the condition on the second derivative Dxy. 
endo4.bmp
 
Fig. 16. On the left. Graphical representation of the coefficient F. On the right. The narrowing of the area of interest (in red) by choosing, among the points selected in the previous step, only the ones whose F is positive on the right side of the face and negative on the left side. 
endo5.bmp
 
Fig. 17. On the left. Graphical representation of the coefficient e. On the right. The extraction of the landmarks minimizing e. 
 

NASION 
In order to localize the nasion, the algorithm firstly identifies a region of interest using the pronasal and the endocanthions, already extracted. Then, it select the points whose Shape Index SÎ [-0.375,0.625] and that are critical points; finally it maximize the principal curvature k1 to extract the landmark. The behavior of the principal curvature is not so evident because the local maximum is not very high in comparison with the other maximums. In Figs. 18–21 are shown the steps of the process. 
n_zona.bmp
 
Fig. 18. Region of interest for the extraction of the nasion. 

nasion1.bmp
 
Fig. 19. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red) by choosing only the points whose Shape Index lies in the range corresponding to the surface of ridge, saddle ridge, saddle point or saddle rut. 
nasion2.bmp
 
Fig. 20. On the left. Critical points, in red. On the right. The narrowing of the area of interest (in red or yellow) by choosing, among the points selected in the previous step, only the critical ones. 
nasion3.bmp
 
Fig. 21. On the left. Graphical representation of k1. On the right. The extraction of the landmark maximizing k1. 

INNER EYEBROW 
Firstly, the process identifies two regions of inte-rest using the coordinates of the endocanthions (Fig. 22), then the areas are narrowed with the condition on the Shape Index (Fig. 23). Finally, the right IE is extracted maximizing the coefficient f, instead the left IE is extracted minimizing f (Fig. 24). 

ie_zona.bmp
 
Fig. 22. Regions of interest for the extraction of the inner eyebrows. 
ie1.bmp
 
Fig. 23. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red) by choosing only the points whose Shape Index is greater than 0.5. 
ie2.bmp
 
Fig. 24. On the left. Graphical representation of f. On the right. The extraction of the landmarks maximizing f for the right IE and minimizing f for the left IE. 

LABRUM SUPERIOR 
The process of extraction of the labrum superior is very simple, using the subnasal it identifies a region   
of interest (Fig. 25), then it narrows the area using the Shape Index (Fig. 26) and it extracts the landmark maximizing the curvedness index C (Fig. 27). 

ls_zona.bmp
 
Fig. 25. Region of interest for the extraction of the labrum superior. 
ch1.bmp
 
Fig. 26. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red) by choosing only the points whose Shape Index lies in the range corresponding to the surface of dome. 
ls2.bmp
 
Fig. 27. On the left. Graphical representation of the curvedness index C. On the right. The extraction of the landmark maximizing C. 

LABRUM INFERIOR 
The first step of the process of extraction of the labrum inferior consists in identifying a region of interest using the subnasal and the labrum superior (Fig. 28). Then, the process is equal to the one of the labrum superior (Figs. 29 and 30). 

li_zona.bmp
 
Fig. 28. Region of interest for the extraction of the labrum inferior. 
li1.bmp
 
Fig. 29. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red) by choosing only the points whose Shape Index lies in the range corresponding to the surface of dome. 
li2.bmp
 
Fig. 30. On the left. Graphical representation of the curvedness index C. On the right. The extraction of the landmark maximizing C. 

CHELION 
The extraction of the chelions is not easy, because most of the geometric descriptors does not have a significant behavior in these points and the discri-minating features that we have identified were not so evident in all the scans. So, the algorithm firstly identifies two regions of interest using the coor-dinates of the labrum superior and inferior and of the alae (Fig. 31). Then, it narrows the area using the condition on the Shape Index (Fig. 32) and it extracts the landmarks maximizing the second derivative Dxy for the right chelion and minimizing Dxy for the left one (Fig. 33). 

ch_zona.bmp
 
Fig. 31. Regions of interest for the extraction of the chelions. 
ch1.bmp
 
Fig. 32. On the left. Graphical representation of the Shape Index. On the right. The narrowing of the area of interest (in red) by choosing only the points whose Shape Index lies in the range corresponding to the surface of rut or cup. 
ch2.bmp
 
Fig. 33. On the left. Graphical representation of Dxy. On the right. The extraction of the landmarks maximizing Dxy for the right CH and minimizing Dxy for the left CH. 

RESULTS 
The 13 resulting landmarks of the nine shells stu-died are shown in Fig. 34. 
In parallel with our extraction, it was asked to four practitioners to take the shells into exam and deter-mine where the true landmarks were located. A set of coordinates of "real landmarks" has been collected for each physician and, although the indicated coor-dinates were most of the times the same, a mean has been computed to state the final real ones. 
The coordinates of landmarks extracted with our algorithm were compared with the real points through a brief statistical study. Euclidian distances between the correct landmarks and the respective points given by our algorithm were computed. The values of these distances are shown in Table 2, where the unit of measurement is the millimetre. 

 
Fig. 34. The extracted landmarks for the nine shells. 
Table 2. Numerical values of the distances between the correct landmarks and the landmarks obtained with the algorithm for the 9 shells. 
 
 PN 
 SN 
 ALAdx 
 ALAsx 
 ENdx 
 ENsx 
 N 
 IEdx 
 IEsx 
 LS 
 LI 
 CHdx
 CHsx 
 
Bart 
 0.91 
 0 
 1.87 
 1.70 
 1.63 
 1.31 
 0.54 
 0.75 
 1.33 
 0.54 
 2.16 
 3.53 
 3.96 
 
Elena 
 0.96 
 1.20 
 0.86 
 0 
 1.85 
 1.94 
 1.12 
 0.96 
 1.74 
 2.73 
 0 
 1.72 
 0 
 
Fede 
 0 
 0.65 
 0.65 
 0.70 
 1.32 
 1.31 
 0.57 
 1.32 
 1.31 
 0 
 0.57 
 0.69 
 2.40 
 
Gian 
 0 
 1.38 
 1.80 
 0 
 2.31 
 1.40 
 0.70 
 0 
 2.39 
 2.12 
 0.91 
 1.73 
 1.10 
 
Gio 
 0 
 0.70 
 0 
 0 
 1.18 
 1.42 
 0.44 
 0.57 
 0.56 
 0.72 
 0 
 2.05 
 0 
 
Lisa 
 0.71 
 1.68 
 0 
 3.04 
 1.52 
 2.20 
 0 
 1.55 
 2.45 
 1.66 
 0.55 
 2.30 
 2.85 
 
Paul 
 0.47 
 0.88 
 0.70 
 1.70 
 0.47 
 0.84 
 2.10 
 0.71 
 0.71 
 0.84 
 1.58 
 0.85 
 0.47 
 
Pie 
 1.29 
 1.11 
 0.61 
 1.03 
 0.93 
 0.47 
 1.11 
 0.94 
 0.61 
 0 
 1.97 
 1.15 
 0.97 
 
Simon 
 0.61 
 1.45 
 0.63 
 0 
 1.05 
 0.61 
 0.52 
 1.24 
 0.53 
 0.80 
 2.07 
 2.36 
 1.58 
 

 

Then, for each shell, sample mean E and sample variance . of these distances di were calculated: 

 

 
where N is the number of landmarks used. The results are shown in Table 3. 
As can be seen from Tables 2 and 3, all errors lie in the range between 0 and 3.5 mm and the mean distance for each shell is in the range between 0.6 and 1.6 mm. It is important to underline that the quality of the results does not depend on the number of weeks of the fetus. For instance, Gio and Gian are two fetuses of 22 weeks, but Gio is the one with the best results while, on the contrary, Gian has one of the highest mean error values. Instead, results are highly depending on echography quality. Lisa, which shows the worst result, has a bad quality because the baby had a hand next to the face, so it was difficult to extract a clear image of the whole face.  
In order to check whether some landmarks were more subject to errors, mean and variance were com-puted also for every landmark. The values are given in Table 4, while the trends of mean and variance were graphically represented in Fig. 35. 
grafico.bmp
 
Fig. 35. Graphical representation of values of sample mean and sample variance computed between the 9 shells for the 13 landmarks. 
The localization seems to be quite accurate. The values show that the position of the pronasal is the most accurate with a mean distance of 0.55 mm. Instead, the least precise landmarks belong to the mouth region, in particular the two chelions have the highest values, with a mean distance of 1.82 mm for the right one and 1.48 mm for the left one. As explained in the section relating to chelions in section 3, the extraction of the two chelions is not trivial, moreover often in these areas the mouth is not well defined. 

Table 3. Values of sample mean and sample variance computed between the 13 landmarks of each shell. 
 
 Bart 
 Elena 
 Fede 
 Gian 
 Gio 
 Lisa 
 Paul 
 Pie 
 Simon 
 
mean 
 1.56 
 1.16 
 0.90 
 1.22 
 0.59 
 1.58 
 0.95 
 0.94 
 1.05 
 
variance 
 1.22 
 0.65 
 0.38 
 0.68 
 0.38 
 0.95 
 0.24 
 0.20 
 0.57 
 

Table 4. Values of sample mean and sample variance computed between the 9 shells for the 13 landmarks. 
 
 PN 
 SN 
 ALAdx 
 ALAsx 
 ENdx 
 ENsx
 N 
 IEdx 
 IEsx 
 LS 
 LI 
 CHdx
 CHsx
 
mean 
 0.55 
 1.00 
 0.79 
 0.91 
 1.36 
 1.28 
 0.79 
 0.89 
 1.29 
 1.05 
 1.09 
 1.82 
 1.48 
 
variance 
 0.22 
 0.26 
 0.44 
 1.14 
 0.29 
 0.33 
 0.36 
 0.21 
 0.58 
 0.88 
 0.76 
 0.77 
 1.84 
 


CONCLUSION 
In this work we have implemented a landmarking algorithm for extracting facial landmarks from nine fetuses at 22–32 weeks' gestation obtained through 3D ultrasound. The attempt was to give a further utility to 3D ultrasound, that nowadays mainly has the role of providing future mothers with a three-dimensional rendered image of their babies. Although some work has been undertaken on this tool, no auto-matic landmarking algorithms for diagnostic purposes have been done yet. This study tries to answer to this lack. 
The method designed to elaborate the algorithm relies on the behaviour that some geometrical des-criptors have on faces, when they are computed point-by-point on facial shells. The results obtained, validated with the support of four practitioners, show that the localization is quite accurate. All errors lie in the range between 0 and 3.5 mm and the mean distance for each shell is in the range between 0.6 and 1.6 mm. The landmarks showing the highest errors are the ones belonging to the mouth region; in particular the two chelions are the least precise with a mean distance of 1.82 mm for the right one and 1.48 mm for the left one. Instead, the most precise landmark is the pronasal with a mean distance of 0.55 mm. 
REFERENCES 
Borrell A, Santolaya-Forgas J, Horbaczewski C, Henry RD, Dunn-Albanese L, Robinson JN (2011). Is the starting section for 3D volume acquisition in the first trimester relevant in the post hoc analysis of aneuploidy screening markers and fetal anatomy? Prenat Diagn 31:1305–10. 
Bromley B, Shipp TD, Benacerraf B (2007). Assessment of the third-trimester fetus using 3-dimensional volumes: a pilot study. J Clin Ultrasound 35:231–7. 
Faure JM, Captier G, Bäumler M, Boulot P (2007). Sono-graphic assessment of normal fetal palate using three-dimensional imaging: a new technique. Ultrasound Obstet Gynecol 29:159–65. 
Johnson DD, Pretorius DH, Budorick NE, et al. (2000). Fetal lip and primary palate: three-dimensional versus two-dimensional US. Radiology 217:236–9. 
Koenderink JJ, van Doorn AJ (1992). Surface shape and curvature scales. Image Vision Comput 10:557–64. 
Lituania M, Tonni G (2013). Bifid uvula and familial Stickler syndrome diagnosed prenatally before the sonographic ‘‘equals sign’’ landmark. Arch Gynecol Obstet 288:483–7. 
Manganaro L, Tomei A, Fierro F, et al. (2011). Fetal MRI as a complement to US in the evaluation of cleft lip and palate. Radiol Med 116:1134–48. 
Martinez-Ten P, Perez-Pedregosa J, Wong AE (2010). First-trimester assessment of the nasal bones using the retronasal triangle View. J Ultrasound Med 29:1555–61. 
Martinez-Ten P, Adiego B, Illescas T, Bermejo C, Wong AE, Sepulveda W (2012). First-trimester diagnosis of cleft lip and palate using three-dimensional ultrasound. Ultrasound Obstet Gynecol 40:40–6. 
Paladini D, Volpe P (2006). Posterior fossa and vermian morphometry in the characterization of fetal cerebellar abnormalities: a prospective three-dimensional ultrasound study. Ultrasound Obstet Gynecol 27:482–9. 
Persico N, Molina F, Borenstein M, Azumendi G, Nicolaides KH (2010). Nasal-bone length in euploid fetuses at 16–24 weeks’ gestation by three-dimensional ultrasound. Ultrasound Obstet Gynecol 36:285–90. 
Pilu G, Segata M, Ghi T, et al. (2006). Diagnosis of midline anomalies of the fetal brain with the three-dimensional median view. Ultrasound Obstet Gynecol 27:522–9. 
Plasencia W, Dagklis T, Sotiriadis A, Borenstein M, Nicolaides KH (2007). Frontomaxillary facial angle at 11 + 0 to 13 + 6 weeks’ gestation – reproducibility of measurements. Ultrasound Obstet Gynecol 29:18–21. 
Pugash D, Brugger PC, Bettelheim D, Prayer D (2008). Prenatal ultrasound and fetal MRI: The comparative value of each modality in prenatal diagnosis. Eur J Radiol 68:214–26. 
Rochelson B, Vohra N, Krantz D, Macri VJ (2006). Geo-metric morphometric analysis of shape outlines of the normal and abnormal fetal skull using three-dimen-sional sonographic multiplanar display. Ultrasound Obstet Gynecol 27:167–72. 
Rotten D, Levaillant JM, Martinez H, Ducou Le Pointe H, Vicaut É (2002). Blackwell Science Ltd The fetal mandible: a 2D and 3D sonographic approach to the diagnosis of retrognathia and micrognathia. Ultrasound Obstet Gynecol 19:122–30. 
Rotten D, Levaillant JM (2004). Two- and three-dimensional sonographic assessment of the fetal face. A systematic analysis of the normal face. Ultrasound Obstet Gynecol 23:224–31. 
Sepulveda W, Wong AE, Martinez-Ten P, Perez-Pedregosa J (2010). Retronasal triangle: a sonographic landmark for the screening of cleft palate in the first trimester. Ultrasound Obstet Gynecol 35:7–13. 
Sepulveda W, Wong AE, Vinals F, Andreeva E, Adzehova N, Martinez-Ten P (2012a). Absent mandibular gap in the retronasal triangle view: a clue to the diagnosis of micrognathia in the first trimester. Ultrasound Obstet Gynecol 39:152–6. 
Sepulveda W, Wong AE, Sepulveda F, Martinez-Ten P, Ximenes R (2012b). Fetal magnetic resonance imaging and three-dimensional ultrasound in clinical practice: General aspects. Best Pract Res Cl Ob 26:575–91. 
Sepulveda W, Ximenes R, Wong AE, Sepulveda F, Martinez-Ten P (2012c). Fetal magnetic resonance imaging and three-dimensional ultrasound in clinical practice: Applications in prenatal diagnosis. Best Pract Res Cl Ob 26:593–624. 
Tonni G, Lituania M (2012). OmniView algorithm – A novel 3-dimensional sonographic technique in the study of the fetal hard and soft palates. J Ultrasound Med 31:313–8. 
Turan S, Turan OM, Ty-Torredes K, Harman CR, Baschat AA (2009). Standardization of the first-trimester fetal cardiac examination using spatiotemporal image corre-lation with tomographic ultrasound and color Doppler imaging. Ultrasound Obstet Gynecol 33:652–6. 
Vinals F, Munoz M, Naveas R, Giuliano A (2007). Trans-frontal three-dimensional visualization of midline ce-rebral structures. Ultrasound Obstet Gynecol 30:162–8. 
Wong HS, Tait J, Pringle KC (2009). Examination of the secondary palate on stored 3D ultrasound volumes of the fetal face. Ultrasound Obstet Gynecol 33:407–11. 
APPENDIX 
The First and Second Fundamental Forms are used to measure distance on surfaces and are defined by 
 , 

 , 

respectively, where E, F, G, e, f and g are their Coefficients. Curvatures are used to measure how a regular surface x bends in . If D is the differential and N is the normal plane of a surface, then the determinant of DN is the product (–k1)(–k2) = k1k2 of the Principal Curvatures, and the trace of DN is the negative –(k1 + k2) of the sum of Principal Curvatures. In point P, the determinant of  is the Gaussian Curvature K of x at P. The negative of half of the trace of DN is called the Mean Curvature H of x at P. In terms of the principal curvatures can be written 


 
 
Some definitions of these descriptors are given.  These are the forms implemented in the algorithm: 
  

 
 

 
 
 
where h is a differentiable function . It is, therefore, convenient to have at hand formulas for the relevant concepts in this case. To obtain such formulas let us parametrize the surface by 

 
 
 
where u = x, v = y. 
The most used descriptors are surely the Shape and Curvedness Indexes S and C, introduced by Koenderink et al. (1992): 

             

 
 
For the role they play in the work, a little digression about their significance is needed. Their meaning is shown in Figs. 36–38 and in Table 5. 

S
 
Fig. 36. Illustration of Shape Index scale divided into seven categories. Different subintervals of its range  [-1,1] correspond to seven geometric surfaces. 
Table 5. Topographic classes. 
Class 
 S 
 Type 
 H 
 K 
 
cup/pit 
 [-1,-0.625) 
 elliptical convex 
 + 
 + 
 
rut/valley 
 [-0.625,-0.375) 
 cylindrical convex 
 + 
 0 
 
saddle rut/saddle valley 
 [-0.375,-0.125) 
 hyperbolic convex 
 + 
 - 
 
saddle 
 [-0.125,0.125) 
 hyperbolic symmetric 
 0 
 - 
 
saddle ridge 
 [0.125,0.375) 
 hyperbolic concave 
 - 
 - 
 
ridge 
 [0.375,0.625) 
 cylindrical concave 
 - 
 0 
 
dome/peak 
 [0.625,1) 
 elliptical concave 
 - 
 + 
 

 

 
C
 
Fig. 37. Curvedness Index scale, whose range is(-., .). 
SC
 
Fig. 38. Indexes (S,C) are viewed as polar coordinates in the (k1, k2)-plane, with planar points mapped to the origin. The effects on surface structure from varia-tions in the curvedness (radial coordinate) and Shape Index (angular coordinate) parameters of curvature, and the relation of these components to the principal curvatures (k1 and k2). The degree of curvature in-creases radially from the centre.