ERK'2020, Portorož, 344-347 344 Semi-automated correction of MOBIUS eye region annotations Oˇ zbej Golob Peter Peer Matej Vitek Computer Vision Laboratory, Faculty of Computer and Information Science, University of Ljubljana, Veˇ cna pot 113, 1000 Ljubljana E-poˇ sta: ozbej.golob@gmail.com,fpeter.peer, matej.vitekg@fri.uni-lj.si Semi-automated correction of MOBIUS eye region annotations MOBIUS is a publicly available dataset of ocular images with manually created annotations of the various eye re- gions. However, manual markups are prone to human error, and in this work we explore the semi-automatic ap- proach we utilised to correct flaws in the MOBIUS dataset annotations. This improves the dataset’s usability for segmentation and training deep segmentation models. The program we wrote removes areas that are outside of the required area (outliers), areas that are inside of other ar- eas (inliers) and missing or blurred edges from annota- tions. This results in fixed annotations and a better deep learning model. Evaluation was performed on a model built on new annotations and a model built on original annotations from MOBIUS dataset. The model trained on fixed annotations achieved significantly better results in all metrics than the one trained on the original data. Thus, for a better learning model, the distortions on orig- inal annotations should be removed. 1 Introduction Ocular biometrics refers to a branch of biometrics that studies the use of various eye parts for recognition. Oc- ular biometrics for recognition is receiving significant at- tention from researchers in recent years. Popular biomet- rics traits are iris, retina, sclera and the periocular region, where iris is the most popular [1, 2]. Sclera biometrics is not as popular as iris or periocu- lar biometrics [3], despite the fact that it has some unique advantages. Sclera is a white region in the eye and con- tains blood vessel patterns that can be used for human recognition [4]. Sclera area is a highly secured part of the eye, so it is impossible to steal. It is a vessel pattern, so it is unique for each individual (even for twins) and it is stable throughout a persons lifetime [5, 6]. Sclera is reliable for recognition in case of potential eye redness and also in the presence of contact lenses [2]. Because of this properties, sclera biometrics is a reliable approach for human identification. Although it is not the most popular type of ocular biometrics, its popularity is rising. This is mainly due to a fact that some interesting recent research was performed on multi-modal eye recognition (using iris and sclera). This research reported that iris information fusion with sclera can increase the applicability of iris biometrics in off-angle and off-axis eye gaze [4]. Recognition systems based on the vasculature of sclera consist of (i) a vasculature segmentation stage, which ex- tracts the vascular structure from sclera and (ii) a recog- nition stage, where the vascular structure is represented with image descriptors, which are then used for recogni- tion. The first stage consists of two steps. The first step locates the sclera and the second step extracts the vascu- lature required for recognition [2]. In the field of sclera segmentation an important and extensive source of approaches and information is the annual Sclera Segmentation Benchmarking Competition (SSBC) [4, 7, 8, 9, 10]. The goal of these competitions was to set a common platform for evaluation of sclera segmentation and to record recent developments in sclera segmentation and recognition in visible spectrum. Before deep learning most segmentation methods used handcrafted features and filter based methods. Only few of them were sclera specific. One example of those meth- ods was Unsupervised Sclera Segmentation (USS) [11]. Most recent research in sclera segmentation uses gen- eral purpose deep models, usually based on convolutional encoder-decoder (CED) architecture. Some of those mod- els are U-Net [2, 12], SegNet [13], ScleraSegNet [14], RITnet [15], RefineNet [16], and DeepLab [17]. Most of the existing research is focused on images taken in the near-infrared (NIR) spectrum, which requires special equipment to capture. The latest research is lean- ing towards images captured in visible spectrum, which can be taken with mobile phone cameras [18]. MOBIUS [19] is currently the only publicly acces- sible dataset for development of deep models for mo- bile ocular biometrics (including sclera segmentation and recognition). Annotations in the dataset are manually produced. Original images are taken with mobile phone cameras and are consequently captured in the visible spec- trum. They also contain sclera, pupil and iris ground truth information, which is especially important for the devel- opment of deep models. Because the annotations were manually produced, a large number of them contain distortions. In this work we present our method that removes distortions from an- notations and fixes them, thus improving the quality of the dataset. This improves the dataset’s usability both for 345 the evaluation of general segmentation methods as well as for learning deep segmentation models. The rest of the paper is structured as follows: in sec- tion 2 we look at the different types of defects in the an- notations and the methods we used to fix them. In section 3 we show both qualitative and quantitative results that point to the impact of our fixes. Finally, we conclude the paper in section 4 with a look at the implications of our work. 2 Methods Our program is written in Python in combination with OpenCV library [20]. The annotations in the dataset con- sist of the sclera (green), iris (red), and pupil (blue). Our program therefore detected red, green and blue contours with the help of the OpenCV functionfindContours. The annotations had three main types of distortions. In this section we look at each of the three types of dis- tortions and the methods used to address them. 2.1 Outliers In theory, only the sclera region should have been green, but some annotations had green spots (outliers) in areas other than the sclera. Outliers appeared with iris and pupil as well. An example of green outliers is shown in Figure 1. To fix this problem with the iris, our program selected the biggest red contour, but only if its center was between the top and bottom of the biggest green contour’s Y co- ordinates and if it had sufficient size (contour area bigger than 2500). Then it did the same for pupil, only with the biggest blue contour. The sclera was somewhat more complex since it wasn’t necessarily one-piece like the iris and pupil were. To address this, the program first selected the biggest green contour, if it had sufficient size (con- tour area bigger than 5000). Then it processed each green contour and selected it only if it partially matched the Y coordinates of the biggest green contour. Figure 1: Annotation with green outliers (best viewed with zoom online). 2.2 Inliers In some annotations there were blue and red spots (in- liers) inside of the sclera region, which was supposed to be green. Inliers appeared on iris and pupil as well. An example of red inliers in the pupil is shown in Figure 2. For iris, the program selected the biggest red contour and filled it with red. It did the same for the pupil, only with the biggest blue contour. For the sclera, program selected all contours that were left after the process of re- moving outliers (see the previous section) and filled them with green. Figure 2: Annotation with red inliers (best viewed with zoom online). 2.3 Blur and missing edges Some annotations had edges between sclera/iris or iris/pupil that weren’t processed correctly. They were either miss- ing or blurred. Example of missing edges between the sclera and iris regions is shown in Figure 3. To fix this issue, our program created a Convex Hull of each green contour. Then it filled those contours with green color. After the process of fixing green missing edges, it created a Convex Hull of the biggest red contour and filled it with red. After that, it filled the biggest blue contour with blue to remove inliers. Figure 3: Annotation with missing edges (best viewed with zoom online). 3 Results In this section we first look at a qualitative comparison between the original annotations from section 2 and the annotations corrected with our method. We also perform a quantitative analysis, using a state-of-the-art deep model for image segmentation, DeepLab [17]. 346 Figure 4 shows an example of how our method re- moved green outliers from a faulty annotation. We can see the original annotation and the processed annotation. We also highlight the differences to clarify just what was removed/added in the fixed annotation. As we can see, the green outlier from the bottom of the before image has been removed. Figure 5 shows images of annotation with red inliers before and after the program processed it and the differ- ences. Notice that the red inliers from the pupil have been removed. Figure 6 shows images of annotation with missing edges before and after the program processed it and dif- ferences. Missing edges between sclera and iris have been removed. Because we filled each green and red contour using their Convex Hulls, the jagged edges present in some an- notations were also smoothed out as an added bonus. To demonstrate the impact of our annotation fixes, we trained and evaluated DeepLabV3+ [17] (which is cur- rently state-of-the-art in many segmentation tasks) in the task of sclera segmentation on both the original annota- tions from the MOBIUS dataset and the annotations that our program fixed. We report the results in the form of several standard metrics in image segmentation: the mean and frequency-weighted intersection-over-union (fwIoU, mIoU), the F 1 -measure and the average precision. In or- der to get a better idea of the data distribution and model generalisation, we report the results as means and stan- dard deviations over all the images in the testing part of the dataset. Table 1: Comparison of accuracy measurements, reported as mean standard deviation over the images in the test dataset. Measurement Old ( ) New ( ) mIoU 0:797 0:159 0:835 0:147 fwIoU 0:919 0:041 0:936 0:031 F 1 -measure 0:771 0:13 0:814 0:119 Average precision 0:755 0:17 0:827 0:12 As we can see from Table 1, the deep model trained and evaluated on the fixed annotations achieved better re- sults with smaller standard deviations in all metrics than the same deep model trained and evaluated on the original annotations. 4 Conclusion Our program successfully removed several different types of distortions present in the manual annotations in the MOBIUS dataset. The evaluation results show the model trained on the corrected annotations achieved better over- all performance (higher metric means), as well as more consistent performance across different images (lower met- ric standard deviations). This implies that the new anno- tations are more consistent across the entire dataset and allow for better training of the model, as well as for bet- ter generalisation to previously unseen images. Our ap- proach therefore resulted in fixed annotations that notice- ably improved the quality of the MOBIUS dataset. References [1] Peter Rot, ˇ Ziga Emerˇ siˇ c, Vitomir ˇ Struc, and Peter Peer. Deep multi-class eye segmentation for ocular biometrics. In 2018 IEEE International Work Conference on Bioin- spired Intelligence (IWOBI), pages 1–8, 07 2018. [2] Peter Rot, Matej Vitek, Klemen Grm, ˇ Ziga Emerˇ siˇ c, Pe- ter Peer, and Vitomir ˇ Struc. Deep sclera segmentation and recognition. In A. Uhl, C. Busch, S. Marcel, and R. Veldhuis, editors, Handbook of Vascular Biometrics, pages 395–432. Springer, 2020. [3] Peter Rot, Matej Vitek, Blaˇ z Meden, ˇ Ziga Emerˇ siˇ c, and Peter Peer. Deep periocular recognition: A case study. In 2019 IEEE International Work Conference on Bioinspired Intelligence (IWOBI), pages 000021–000026. IEEE, 2019. [4] Abhijit Das, Umapada Pal, Miguel Ferrerc, and Michael Blumensteina. Ssbc 2015: Sclera segmentation bench- marking competition. pages 1–6, 09 2015. [5] Matej Vitek, Peter Rot, Vitomir ˇ Struc, and Peter Peer. A comprehensive investigation into sclera biometrics: A novel dataset and performance study. Neural Computing & Applications, pages 1–15, 2020. [6] V . Patil and A. M. Patil. Human identification method: Sclera recognition. International Journal of Computer Science and Network (IJCSN), 6(1):24–29, 2017. [7] Abhijit Das, Umapada Pal, Miguel Ferrer, and Michael Blumenstein. Ssrbc 2016: Sclera segmentation and recog- nition benchmarking competition. pages 1–6, 06 2016. [8] Abhijit Das, Umapada Pal, Miguel Ferrer, Michael Blu- menstein, Dejan ˇ Stepec, Peter Rot, ˇ Ziga Emerˇ siˇ c, Peter Peer, Vitomir ˇ Struc, S V Aruna kumar, and B S Harish. Sserbc 2017: Sclera segmentation and eye recognition benchmarking competition. pages 742–747, 10 2017. [9] Abhijit Das, Umapada Pal, Miguel Ferrer, Michael Blu- menstein, Dejan ˇ Stepec, Peter Rot, ˇ Ziga Emerˇ siˇ c, Peter Peer, and Vitomir ˇ Struc. Ssbc 2018: Sclera segmentation benchmarking competition. pages 303–308, 02 2018. [10] Abhijit Das, Umapada Pal, Michael Blumenstein, and zhun sun. Sclera segmentation benchmarking competition in cross-resolution environment. 06 2019. [11] Daniel Riccio, Nadia Brancati, Maria Frucci, and Diego Gragnaniello. An Unsupervised Approach for Eye Sclera Segmentation, pages 550–557. 01 2018. [12] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image seg- mentation. volume 9351, pages 234–241, 10 2015. [13] V . Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for im- age segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12):2481–2495, 2017. [14] C. Wang, Y . He, Y . Liu, Z. He, R. He, and Z. Sun. Scle- rasegnet: an improved u-net model with attention for ac- curate sclera segmentation. In 2019 International Confer- ence on Biometrics (ICB), pages 1–8, 2019. [15] Aayush K Chaudhary, Rakshit Kothari, Manoj Acharya, Shusil Dangi, Nitinraj Nair, Reynold Bailey, Christopher 347 Figure 4: Annotation with green outliers (best viewed with zoom online). (a) Original annotation. (b) Fixed annotation. (c) Differences. Figure 5: Annotation with red inliers (best viewed with zoom online). (a) Original annotation. (b) Fixed annotation. (c) Differences. Figure 6: Annotation with missing edges (best viewed with zoom online). (a) Original annotation. (b) Fixed annotation. (c) Differences. Kanan, Gabriel Diaz, and Jeff B Pelz. Ritnet: Real-time semantic segmentation of the eye for gaze tracking. arXiv preprint arXiv:1910.00694, 2019. [16] Guosheng Lin, Anton Milan, Chunhua Shen, and Ian Reid. Refinenet: Multi-path refinement networks for high-resolution semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. [17] Liang-Chieh Chen, George Papandreou, Iasonas Kokki- nos, Kevin Murphy, and Alan L. Yuille. Deeplab: Se- mantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. CoRR, abs/1606.00915, 2016. [18] Ishan Nigam, Mayank Vatsa, and Richa Singh. Ocular biometrics: A survey of modalities and fusion approaches. Information Fusion, 26:1 – 35, 2015. [19] Matej Vitek, Abhijit Das, Yann Pourcenoux, Alexan- dre Missler, Calvin Paumier, Sumanta Das, Ishita De Ghosh, Diego R. Lucio, Luiz A. Zanlorensi Jr., David Menotti, Fadi Boutros, Naser Damer, Jonas Henry Grebe, Arjan Kuijper, Junxing Hu, Yong He, Caiy- ong Wang, Hongda Liu, Yunlong Wang, Zhenan Sun, Daile Osorio-Roig, Christian Rathgeb, Christoph Busch, Juan Tapia Farias, Andres Valenzuela, Georgios Zam- poukis, Lazaros Tsochatzidis, Ioannis Pratikakis, Sabari Nathan, R Suganya, Vineet Mehta, Abhinav Dhall, Ki- ran Raja, Gourav Gupta, Jalil Nourmohammadi Khiarak, Mohsen Akbari-Shahper, Farhang Jaryani, Meysam Asgari-Chenaghlu, Ritesh Vyas, Sristi Dakshit, Sagnik Dakshit, Peter Peer, Umapada Pal, and Vitomir ˇ Struc. SSBC 2020: Sclera segmentation benchmarking compe- tition in the mobile environment. In IEEE International Joint Conference on Biometrics (IJCB), 2020. [20] G. Bradski. The OpenCV Library. Dr. Dobb’s Journal of Software Tools, 2000.