LIVE VIDEO FACE DETECTION AND RECOGNITION Jalwinder Pingal B.Tech, Kurukshetra University, 2004 PROJECT Submitted in partial satisfaction of the requirements for the degree of MASTER OF SCIENCE in COMPUTER ENGINEERING at CALIFORNIA STATE UNIVERSITY, SACRAMENTO SPRING 2010 LIVE VIDEO FACE DETECTION AND RECOGNITION A Project by Jalwinder Pingal Approved by: __________________________________, Committee Chair Suresh Vadhva, Ph.D. __________________________________, Second Reader Preetham B. Kumar, Ph.D. ____________________________ Date ii Student: Jalwinder Pingal I certify that this student has met the requirements for format contained in the University format manual, and that this project is suitable for shelving in the Library and credit is to be awarded for the Project. __________________________, Graduate Coordinator Suresh Vadhva, Ph.D. Department of Computer Engineering iii ________________ Date Abstract of LIVE VIDEO FACE DETECTION AND RECOGNITION by Jalwinder Pingal Face recognition is one of the most important fields of the modern biometric applications. Face recognition system uses two sub-systems named face detection system and image database system. Face recognition can be of feature based and image based. Feature based method uses features like skin color, eyes, nose and mouth to detect and recognize human face whereas image based method utilizes some preprocessed image sets for detection [1]. The project implements feature based face recognition system which first finds any face or faces in the color image and then matches it against the database to recognize the individuals [2]. Here, the skin color pixels are used to filter out the interesting regions of human skin from other non- interesting regions. Once the skin regions are located, facial features like mouth, eyebrow and nose are extracted to locate the human face. Then, the detected face from image will be compared with the database of training images to find a match. The project also uses the videos from web cam and the video file database to locate human faces. The live video face detection framework works on each and every frame of the video to implement the detection system. The recognition of human beings is also being performed on the video stream coming from either store video or live video from web-cam. _______________________, Committee Chair Suresh Vadhva, Ph.D. _______________________ Date iv ACKNOWLEDGEMENTS First of all I want to express my deepest thanks to Dr. Suresh Vadhva who permitted me for this project under his supervision. He provided me the guidance and supervision needed for the successful completion of the project. Despite of his busy schedule, he always gave his support and help at all the times. I would also like to thank Dr. Preetham Kumar for being the second reader and providing his valuable suggestions on the project report. In addition, I am also thankful to the faculty of Computer Science and Electrical Engineering in helping me achieving my Master degree from the university. Last, but not least, I am very thankful to my family in supporting and encouraging me throughout my education. Without their motivation and support, it would not have been possible for me to complete my education. v TABLE OF CONTENTS Page Acknowledgements ............................................................................................................. v List of Figures .................................................................................................................... ix Chapter 1. INTRODUCTION .......................................................................................................... 1 2. CHALLENGES AND VIDEO/IMAGING CONCEPTS ............................................... 5 2.1 Challenges in Face Recognition................................................................................. 5 2.1.1 Difference in Lighting Conditions ........................................................................ 5 2.1.2 Skin Color Variations ........................................................................................... 6 2.1.3 Variation in Face Angle or Orientation Variation ................................................ 6 2.2 Important Concepts of Video and Imaging Applications .......................................... 6 2.2.1 Image Processing .................................................................................................. 6 2.2.2 Color Space ........................................................................................................... 7 2.2.3 RGB Color Space .................................................................................................. 7 3. SOFTWARE TOOLS USED .......................................................................................... 9 3.1 Java Programming Language ..................................................................................... 9 3.2 Java Advanced Imaging (JAI) ................................................................................. 10 vi 3.3 Java Media Framework (JMF) ................................................................................. 10 3.4 Java Development Kit (JDK) ................................................................................... 11 3.5 NetBeans IDE .......................................................................................................... 11 4. IMAGE FACE DETECTION ....................................................................................... 13 4.1 Reading the Image ................................................................................................... 15 4.2 Implement Feature-Based Algorithm ...................................................................... 17 4.2.1 Identify Pixels of Interest .................................................................................... 17 4.2.2 Edge Detection .................................................................................................... 17 4.2.2.1 Laplacian Edge Detection ............................................................................... 18 4.2.2.2 Gradient Edge Detection ................................................................................. 18 4.2.2.2.1 Sobel Operator ............................................................................................ 19 4.2.3 Region Grouping ................................................................................................. 21 4.2.4 Facial Feature Extraction .................................................................................... 21 4.2.4.1 Eyebrows Detection ........................................................................................ 22 4.2.4.2 Nose Detection ................................................................................................ 23 4.2.4.3 Mouth Detection ............................................................................................. 24 4.3 Marking the Face with Boundary ............................................................................ 25 5. FACE RECOGNITION ON STILL IMAGES ............................................................. 26 vii 6. FACE DETECTION FROM VIDEO ........................................................................... 30 6.1 Face Detection On the Video From Video File ....................................................... 30 6.2 Face Detection On Live Video ................................................................................ 32 7. FACE RECOGNITION ON VIDEO ............................................................................ 34 8. IMPLEMENTATION RESULTS ................................................................................ 40 9. CONCLUSION ............................................................................................................. 41 Bibliography ..................................................................................................................... 42 Attachments ...................................................................................................................... 44 viii LIST OF FIGURES Page Figure 2.2.3: RGB Color Model [9] ................................................................................... 8 Figure 4.1.1: Project Run Output Window ....................................................................... 15 Figure 4.1.2: Loaded Image and The Path Selection ........................................................ 16 Figure 4.2.2.2.1: Value of Convolution Masks [19] ......................................................... 19 Figure 4.2.2.2.2 Results of The Edge Detection ............................................................... 20 Figure 4.2.4.1 Eyebrows Detection................................................................................... 22 Figure 4.2.4.2: Nose Detection ......................................................................................... 23 Figure 4.2.4.3: Mouth Detection ....................................................................................... 24 Figure 4.3: Marking The Face With Boundary................................................................. 25 Figure 5.1: Selected Image and The Database With The Selected Image ........................ 27 Figure 5.2 Output After Recognition Triggered ............................................................... 28 Figure 5.3: No Matching Output Result ........................................................................... 29 Figure 6.1: Face Detection On The Video From Video File ............................................ 31 Figure 6.2: Face Detection On Web-Cam (Live) Video ................................................... 32 Figure 7.1: Video Face Recognition On Person 1(Sam)................................................... 35 Figure 7.2: Video Face Recognition On Person 2(Amy) .................................................. 36 ix Figure 7.3: No Matching Results On Video File’s Video ................................................ 37 Figure 7.4: Web-Cam Recognition Results ...................................................................... 38 Figure 7.5: Live Video Face Recognition With No Matching Results ............................. 39 x 1 Chapter 1 INTRODUCTION Face recognition has gained substantial attention over in past decades due to its increasing demand in security applications like video surveillance and biometric surveillance. Modern facilities like hospitals, airports, banks and many more another organizations are being equipped with security systems including face recognition capability. Despite of current success, there is still an ongoing research in this field to make facial recognition system faster and accurate [3]. The accuracy of any face recognition system strongly depends on the face detection system. The stronger the face detection system the better the recognition system would be. A face detection system can successfully detect human face from a given image containing face/faces and from live video involving human presence. The main methods used in these days for face detection are feature based and image based. Feature based method separates human features like skin color and facial features whereas image based method used some face patterns and processed training images to distinguish between face and non faces. Feature based method has been chosen because it is faster than image based method and its’ implementation is far more simplified [1] [4]. Face detection from an image and live video is achieved through image processing. Locating the faces from images is not a trivial task; because images not just contain human faces but also non-face objects in clutter scenes. Moreover, there are other issues in face recognition like lighting conditions, face orientations and skin colors. Due to these reasons, the accuracy of any 2 face recognition system cannot be 100%. The objective of this project is to implement a face recognition system which first detects the faces present in either single image or multiple video frames; and then identifies the particular person by comparing the detected face with image database. The report is organized in 6 chapters. With introduction already being explained in chapter 1, chapter 2; Section 2.1 focuses on the potential issues in the any face recognition system in the form of difference in the lighting conditions in which the same picture appears differently and the variations in skin color and pose. Explaining chapter 2 further in section 2.2 we will discuss about the important concepts of video and Imaging applications. In any image and video processing application it is important to understand the image processing concepts and the color space in which the current application is being implemented. With the good knowledge of these concepts the image pixels can be modified in a way suitable for the need of any application. Chapter 3 talks about the software tools used in the project mainly related to Java programming language. These tools are Java Advanced Imaging (JAI), Java Media Framework (JMF), Java Development Kit (JDK) and NetBeans IDE. JAI, JMF and JDK are so significant and flexible in the usage that any video and imaging application can be implemented with the ease. It is not wrong to say that any multimedia application cannot be implemented without these tools. 3 In chapter 4 the face detection from image is explained. Chapter 4 mentions the steps used in the project in reading an image to final face detection with boundary around the face. Here, the image is being read from source and then the face detection algorithm is being applied to identify the presence of human being. In the face detection algorithm, the image has to go for number of steps like skin detection, edge detection, region grouping and the features extraction before the face is detected and marked with rectangular boundary. After the face is located in the image we will identify this face. Chapter 5 explains about the implementation of recognize the face or faces present in the image by comparing it with the database. Here, the images are already stored in the database. When any face is detected then we compare it with the images present in the database. If there is a match then recognize returns as true otherwise false. When the detection and recognize on single image is implemented the next advancement is to do the same operation on videos. The detection on the video is implemented in chapter 6. Here the video is coming from stored video file and the live video from web-cam. The frames in either case are buffered and feed it to the image face detection system. If there is any face present in the current frame then it is marked with the boundary and injected back to the display. If there is no face present the video then the unmodified frame is sent back to the display. In this way the video file is being played at the output with modified and unmodified frames. Next is to recognize the human being present in the video file. This part is described in chapter 7. The recognition on video is similar in concept wise to that of 4 image recognition; but difficult in implementation due to continues frames keep coming in video. Here, whoever is present in the video will get recognized if his/her image is present in the database otherwise not. Chapter 8 mentions the results of the implementation of various parts of the project. Then we conclude the project by providing the future work that can be done for future applications in chapter 9. 5 Chapter 2 CHALLENGES AND VIDEO/IMAGING CONCEPTS This chapter focuses on the ongoing confronts in the field of the recognition and some basic concepts of image and video applications. These challenges and the imaging concepts are described in detail in sections 2.1 and 2.2 respectively. 2.1 Challenges in face recognition Over the years, face recognition has gained rapid success with the development of new approaches and techniques. Due to this success, the rate of successful face recognition has been increased to well above 90%. Despite of all this success, all face recognition techniques usually suffer from common challenges of image visibility. These challenges are lighting conditions variations, skin variations and face angle variations [5]. The challenges are explained in descriptive manner below. 2.1.1 Difference in Lighting Conditions The lighting conditions in which the pictures are taken are not always similar because the variations in time and place. The example of lighting variations could be the pictures taken inside the room and the pictures taken outside. Due to these variations, a same person with similar facial expressions may appear differently in different pictures. As a result, if the person has single image store in the database of face recognition, matching could be difficult with the face detection under different lighting conditions [6]. 6 2.1.2 Skin Color Variations Another challenge for skin based face recognition system is the difference in the skin color due to the difference in the races of the people. Due to this variation, sometimes the true skin pixels are filtered out along with the noise present in an image. In addition, the false skin background/non-background is not completely filtered out during noise filtering process. So, it is a tough task is to choose the filter which will cover the entire skin tones of different people and kick out false skin noise [5]. 2.1.3 Variation in face angle or Orientation variation The angle of the human face from camera can be different in different situations. A frontal face detection algorithm can’t work on non-frontal faces present in the image because the geometry of facial features in frontal view is always different than the geometry of facial features in non-frontal view. This is why orientation variation remains the difficult challenge in face detection system [6]. 2.2 Important concepts of video and Imaging applications 2.2.1 Image Processing Image processing is a method of processing the image values, more precisely, the pixels in case of digital images. The purpose of image processing is to modify the input image such that the output image may change parametrically such as in colors and 7 representation. Image processing is the basic part of the face recognition involving digital images. The processing can change the image representation from one color space to another color space. It can also assign different color values to targeted pixels for the purpose of keeping areas of interest in output image. Image processing is also used to increase or decrease image brightness, contrast and other morphological operations [7]. 2.2.2 Color Space Color space is the representation of image colors in two or more color components. Typical examples of color spaces are RGB, YCbCr and HSI color spaces. In each of these color spaces the color of a pixel at any point in an image is the combination of three color components. These color components vary from 0 to maximum value and this maximum value depends on the bits per pixel. The different values in the range give different colors from black (0) to white (255) with 8 bits per pixel. [8]. 2.2.3 RGB color Space RGB color space is the combination of red, green and blue color components. For the 24 bits per pixel, the range of R, G and B varies from 0 to 255. If R, G and B are all 0 then the resulted color will be black. If R, G and B are all 255 then the output color will be white. The concept of the RGB color space is specified in figure 2.b.3. Here the x-axis represents blue color range, y-axis represents green color range and Z-axis represents red color range. As explained above, we can see that black color is 8 represented at the origin and white color is represented at the other corner where red, green and blue are 255 each. Similarly we can have other color values at different corners of cube corresponding to different RGB values [9]. Figure 2.2.3: RGB color model [9] 9 Chapter 3 SOFTWARE TOOLS USED The face detection algorithm is implemented in the Java programming language. The front-end and the back-end parts consist of the reading the image and the displaying output results are also implemented in Java. The platform used to accomplish the project is NetBeans IDE which is the framework to develop the multimedia applications in Java. The explanation of Java and the supporting tools are explained below: 3.1 Java Programming Language Java is an object oriented programming language developed by SUN Microsystems. JAVA contains object and classes need to develop any simple to complex applications with much of ease as comparison to C and C++ programming language. Java inherits the programming syntax from C and C++ languages. Java applications are comprises of classes each corresponding to the particular task and can have relationship among each other with the properties of inheritance. It is a platform independent language means it can be run on any architecture with the help of Java Virtual Machine. By that means the Java code is compiled by the Java Virtual Machine and converts it to the Java byte code. Java virtual machine does the similar task of language specific compiler but the compiled output is such that it can be run on any Java supported platform. The drawback of Java byte code compiled programs is that these are slower in running environment as compared to specific compiler complied programs. To run Java 10 specific applications on any machine Java Runtime Environment (JRE) has to be installed on it. For the development of any application Java has some other associated and supported tools. These tools are named as Java Advanced Imaging, Java Media Framework and Java Development Kit [10]. The following is the description of these tools:3.2 Java Advanced Imaging (JAI) Java Advanced Imaging is a Java supported application programming interface (API) that provides the advanced libraries of image processing. For all the image processing tasks the programmer includes the JAI reference to use built in library functions. Since JAI contains all the image processing routines the developer is not required to purchase additional image processing software while working in the Java environment. With that being said, the usage of JAI is free of charge from SUN Microsystems [11]. Java advanced imaging package also contains encoding and decoding classes for image manipulation. Various popular image formats such as JPG, BMP and PNG are supported in these codec classes [12]. 3.3 Java Media Framework (JMF) Java Media Framework is the additional tool provided with Java to enable the use of audio, video and other time critical multimedia files. JMF performs all the operations from capturing the media file to outputting the file with or without the processing. Java media framework receives media data from several sources such as camcorder and stores 11 it temporarily before processing. It then performs the application required operations such as transcoding, streaming and visualization effects to media file. The output of JMF can either be displayed on display device or can be stored on a disk [13]. 3.4 Java Development Kit (JDK) Java development kit is the Java software development kit containing various software tools necessary to build Java applications. The tools are Java loader, Java compiler, Java debugger, applet viewer and many other development tools. The Java loader is used to load Java programming. Java complier converts hand typed code to machine specific language. Debugger debugs the code step by step to find any code malfunction or the output/value each line of code may have. Without installing JDK the programmer cannot install any framework for developing Java applications [14]. 3.5 NetBeans IDE NetBeans is Integrated Development Environment (IDE) which is written in Java programming language and use for creating Java contained desktop and multimedia applications. NetBeans is fully able to support Java Development Kit features. The advantage of using the IDE is to provide full support to user by saving the time in terms of accessing data and apply setting for any application being developed in Java. The Graphical User Interface (GUI) of NetBeans makes it easy to view, add and delete from application being run on the platform and its debug features are able to find and locate the errors and their causes in the lines of code. NetBeans has several tools and available 12 features necessary to build applications with ease including the editor of source code, graphical user interface and the support for many databases. Source code editor has the support of text styles and background colors, line indentions, text matching, previous saving, code changes features and other class supported features. Graphical User Interface is the visual screen in NetBeans which has the support for current and other forms, buttons and labels with zoom ability, internet connection support, installing and applying features and other important features. NetBeans successfully works with databases such as SQL, Cloudscape, Oracle and Microsoft SQL server. It has also the ability to view and edit the data stored in these databases. It has the provision for writing and executing the databases commands. Just like other language editors NetBeans open the saved project or create new project for Java based and web based applications. It shows the loaded project and its classes (files) in the form of the tree and sub trees. In the code editor the code that we write is being highlighted with the help description for the completion of it. One interesting feature of NetBeans is the called refactoring which is the way to arrange the user written code in such a way so that it is easier to read and modify [15]. 13 Chapter 4 IMAGE FACE DETECTION Face detection is the process of finding the human face from image and, if present, returning the location of it. It is the special case of the object detection. Objects can be anything present in the image including human and non human things like trees, buildings, car and the chair. But, other than a human being itself objects are least likely utilized in advanced applications. So, to find and locate the human face in the image is an interesting and important application of modern time. However, to locate the face is not an easy task in the image since images do not contain only faces but other objects too. Moreover, some of the scenes’ are very complex and to filter out the unwanted information is remain the tough task. Face detection can work on a single image and more than one image in the form of the video. When the presence and the location of the face is found than this information is utilized to implement more sophisticated applications such as recognition and video surveillance implementation. Hence, the success of these applications depends heavily on the detection rate of image face detection system [16]. Face detection from an image can be done by two methods named image based and feature based. Image Based methods treats the whole image as the group of patterns and so each region is classified as the face or non-face. In this method a window is scanned against the parts of the image. On every scan the output is computed and the threshold value is compared with the output value of the window on 14 current part of the image. If it is above than the threshold value then that current part of the image is considered as the face. The size of this window is fixed and chosen according to some experiments with the help of training images size. The advantage of image based methods is the higher percentage of face detection hit rate; however it is slow in terms of computation as compared to feature based methods. Eigenfaces and the neural networks are the two examples of image based approach. In feature based methods features such as skin color, eyes, nose and mouth are separated first from the rest of the image regions. With the extraction of these features the non-interested regions of the image are not required to process further and therefore the processing time is significantly reduced. In feature based methods the skin color pixels are separated first because the color processing is faster which results in the separation of other features. The advantage of the feature based methods is the fast results but less accurate than image based methods. Another advantage is the ease of implementation in real time applications [1]. In the project, the feature based method is implemented. The initial step is to get the image as input and then feature based algorithm is applied to detect the face or faces. Once the face region is determined the next is to mark the boundary around it. The detailed operation of image face detection is explained by following steps: 15 4.1 Reading the image Reading the image is basically loading the image into the face detection framework. For the project, the image is to be read from the computer disk. Figure 4.1.1: Project run output window There are also a collection of frames of video which are captured from web Cam and feed as input to live video face detection framework. The frames are store temporarily in the buffer before being further processed. We can also load the chunk of video stored in the hard disk to video face detection framework. When the project is going to be run the generated output window will appear which has the provision to open the location of the JPEG image file as shown in the figure 4.1.1. 16 Now we select the location of the image by “Browse File” button and the image is loaded in the face detection interface from the selected location. The location of the path and the loaded image is shown in the figure 4.1.2. The filter is applied to choose the image of JPEG type. Other types of images can also be selected by providing the modification in the implementation. Figure 4.1.2: Loaded image and the path selection 17 4.2 Implement feature-based algorithm After reading the image, the project implements number of steps on the image which extract the facial features necessary to locate the human face. It includes the removal of the non-skin like pixels, removal of the false skin pixels and to identify other features like nose, mouth and eyes. The explanation of these steps is as follows: 4.2.1 Identify pixels of interest For any feature based face detection approach; our first step is to identify the skin regions from the image. This is necessary because only skin color is going to be utilized for further image processing. As mentioned earlier an image can contained one or more objects and so we process our image in such a way that non human regions should filter out from skin containing regions. The remaining regions in the image are the likelihood of the skin pixels and skin pixels are the possibility of the facial features. In the process, we convert our colored image to gray-scale image. After converting to gray scale image, skin like regions can be separated from other regions because skin regions are brighter than other regions of the image. This process leaves only skin likelihood regions by removing other non skin regions present in the image [17]. 4.2.2 Edge Detection Skin Identified pixels in last step are pure skin pixels if the image has only skin regions. Since image may contain another skin like objects including skin like 18 background color; there is a strong chance that we may end up getting the false skin regions. These impure skin pixels need to be removed first before further processing. These non skin pixels are removed by the use of the edge detection. Edge detection is a process of marking the boundaries of the objects present in the image by finding the differences in the intensities of the pixels. Due to the differences in the brightness of the pixels there are discontinues path exists from one object present in the image to another. Edge detection helps in image processing by removing the noise like impure skin regions and thus saves incredible time in further processing of the image. Edge Detection can be performed by several methods and are categorizing in two main categories: Laplacian and gradient [18]. These methods are explained as follows: 4.2.2.1 Laplacian Edge Detection In laplacian edge detection the signal undergoes first derivation and then second derivation with respect to time. When second derivation is performed then the signal drops down to zero and below values. Here only zero values are marked to accomplish edge detection [18]. 4.2.2.2 Gradient Edge Detection In this method, also the first derivative of the signal is calculated with respect to time and the maximum value is compare against some threshold value to find the possibility of the image. This category includes methods like Sobel, Roberts and Prewitt 19 operators. Sobel is most widely used method because it is fast and the edge detected image has clear and connected edges [18]. 4.2.2.2.1 Sobel Operator Sobel operator calculates the gradient of the pixel intensity by applying the convolution filters in two directions. These convolution filters are 3* 3 matrices and are used to calculate the gradient in X and Y direction. Two convolution masks are used because our image is two dimensional and therefore one convolution is applied for rows and another convolution is applied for columns. These masks are called Gx and Gy. The formula and the values of Gx and Gy are as follows: The values of convolution masks are given below: Figure 4.2.2.2.1: Value of convolution masks [18] Gradient magnitude is given by the following formula: 20 The approximate value of this magnitude is calculated by formula given below: |G| = |Gx| + |Gy| The result of the edge detection is displayed in the figure 4.2.2.2.2 which shows the boundaries in the image due to the differences in the pixel intensities of the objects present in the image. Due to discontinuities in the image the noise can be removed to eliminate the false face detection and reduce the image processing time. Figure 4.2.2.2.2: Results of the Edge Detection 21 4.2.3 Region Grouping Edge detection eliminated the noise in the image and thus image now has only pure skin regions. These regions could be hand, arm, leg and face. These regions can be overlapped with each others in most of the scenes so it is essential to separate these regions from each other. This task is also called region grouping. Region grouping is performed by connected-neighborhood or connectivity analysis method. In connectivity analysis method a pixel is compared with 8 of its neighbored pixels ; and if any of these pixels has the same value as of the original pixel then these pixels belong to same region otherwise these are from another region. By using this method all of the skin regions are separated from each other [19]. 4.2.4 Facial Feature Extraction When all of the skin regions are separated then we look for the facial features in each and every skin region. The possible facial features are mouth, eyebrows and nose. So we process our regions to extract these facial features. Out of different skin regions, only one region is of our interest which is the possible face region and only this region has facial features. With the proper information on distances of facial features, the facial features can be detected. The process used to identify facial features is explained below: 22 Figure 4.2.4.1: Eyebrows Detection 4.2.4.1 Eyebrows Detection In every region we will look for two small curves separated by certain pixels and in different color to skin region. The color of eyebrows can either be lighter or darker than the skin color. The detection system will look for these two uniform regions and if found than mark these eyebrows with the boundary. The result of the eyebrow detection is shown in the figure 4.2.4.1 where the image is loaded first into interface and then the same image is displayed with the small rectangular boundary on each of the eyebrow [20]. 23 Figure 4.2.4.2: Nose Detection 4.2.4.2 Nose Detection We have already detected the eyebrows. Next is to locate and mark the nose in the facial region. Here the distance between the centers of the starting of eyebrows are taken to determine the location of the nose. The nose is parallel to the eyebrows and at a perpendicular distance to it. The hit rate of nose detection can also be increased by searching of the pixels at the bottom of nose in non-skin color. The black shadow under 24 the nose holes and the curve shape is beneficial to achieve better results. The output of the nose detection interface is shown in the figure 4.2.4.2 [20]. Figure 4.2.4.3: Mouth Detection 4.2.4.3 Mouth Detection The determination of the eyebrows and the nose helps in determining the location of the mouth. The distance in pixels from the nose and eyebrows is taken to find the possibility of the mouth. Since the lips are also in the curve shape, the gradient variation region under the nose is searched to find the location of the mouth. If the person is 25 smiling, the teeth region can be eliminated in the skin detected image with the black pixels. Here the average approximations of lips pixels are considered to find the location of mouth. Figure 4.2.4.3 is visualizing the results of the mouth detection [20]. Figure 4.3: Marking the face with boundary 4.3 Marking the face with boundary Once all of the features are detected on the facial region, the other skin regions are eliminated. Now that the face region is detected, next is to mark the boundary around it. To accomplish this, a rectangle is drawn around the detected face. As shown in the figure 4.3 the original image is displayed with the white color rectangle around the face. 26 Chapter 5 FACE RECOGNITION ON STILL IMAGES Recognition is the process of identifying the particular person present in the image. Recognition is the most powerful and interesting application of image processing and has gained rapid success for over 30 years. Basically, the face detection system locates the face present in the image and after comparing with the database, if present, the identity of that person is revealed. For this purpose, sample images are already stored in the database. The location of the database is set in the recognition system so that each and every image present in the database gets searched. If there is a match then the image is taken from the database and display at the output along with the face detected image. In addition, the name of the matching person will also be displayed. On the other hand, if there is no hit then image is not displayed and a no match message is rather displayed. The method used in the recognition is the Image Search method. It simply takes an input image where a face was found and a directory of images as input. A signature is calculated for the input image. Then for each image in the search directory a signature is also calculated. Basically, the digital signature attempts to assign a unique value to each image based on the contents of the image. A distance is then calculated for each image in the search directory in relation to the input image. The distance is the difference between the signature of the source image and the signature of the image it is being checking 27 against. The image with the smallest distance is returned as the matching image and displayed at the output. Figure 5.1: Selected Image and the database with the selected Image The results of the face recognition on still image are displayed in the figures 5.1 and 5.2. As we can see in figure 5.1 the image is loaded from the database. The location of the database from which the picture is selected is also selected by using “choose search folder” button. 28 Figure 5.2: Output after recognition triggered The next step is to trigger the recognition system to see if the recognition implement successfully. So, as shown in the figure 5.2, we click on detect and search button to initiates the recognition process. Now, the system will search the database against the selected picture to determine if there is any identical image present in database. If present, the matched image is displayed at the output with the rectangular 29 boundary around both of the images. The name of the matching person is also displayed at the output. Figure 5.3: No matching output result To show the output of the face recognition with no matching results, we are using the figure 5.3 where the picture is selected from one folder and the database is selected as the different folder. When the recognition is initiated the system searches the database against the selected picture. Since the selected image is not present in the searched folder, no recognition takes place and instead “Nobody was identified as a match” massage displayed at the output 30 Chapter 6 FACE DETECTION FROM VIDEO Since the face detection on the still images has been implemented successfully, we will move into the next advanced step of locating and marking the face regions on the video stream. The operation on the frames of video is much difficult as compared to the single image. It is because the frames keep coming from the video source and the objects present in the image are contained in more than frame for visibility. In this project, we have two video sources, video file stored on the hard disk of computer and live video coming from the connected Web-cam. In both of the cases the process is to control continues frames and pass these frames to the face detection system. The explanation of both the cases is explained below:6.1 Face detection on the video from video file Here, the path of video file is selected according to the location of the video already stored in the memory. The video file will be open and started using the Java Media Framework. Using the FrameAccess process, the video will be played in the face detection system where FrameAccess acts as a processor to handle each individual frame. This process also instantiates the pass thru codec which is PreAccess Codec and also the PostAccess codec to intercept the data flow. To acquire each video frame before it is modified to the screen, the pass thru codec is used. With the pass through codec, we attach the callback to every frame of the video being viewed. This codec is call every 31 time the frame is changing in the video file. Once the codec operation is done the control will be returned to the player. In order to perform facial detection each frame is intercepted before it was rendered and in the event that a face was detected injects the altered frame back into the video stream. The pass thru codec is also called PreAccess codec. Figure 6.1: Face detection on the video from video file In PreAccess codec two buffers are used named Buffer in and Buffer out. Buffer in is the incoming frame, Buffer out is the next frame to display. PreAccess Codec calls facial detection system on each frame of the video. If a face is found in the individual frame, then the output buffer will be given the modified image with rectangles drawn 32 around the face to be displayed at the output. On the other hand, of no face/faces found then the input buffer is not changed and output Buffer is set to this unmodified input Buffer. It then passes control back to the player which displays the output Buffer. PostAccess Codec is also included to implement the callback to access each frame of the video after it is rendered to the screen. In this manner the video file will be selected in the face detection system. Then the detection will be triggered to display the video at the output with the boundary around each face if present in the video stream. The result of the output of the face detection from the video file is shown in the figure 6.1. Here the location of the video file is selected. When the “Video File” button is clicked then the video from selected video file will be played. If any face is present in the frames then it will be displayed with the boundary around it. 6.2 Face detection on live video Figure 6.2: Face detection on Web-Cam (Live) video 33 The concept of processing of live video frames is same as that of processing the frames on stored Video file. In this case the video is being taken directly from the web-cam. The web-cam connected to the computer is configured in the face detection system using the JMStudio which is the part of the Java Media Framework. Again, upon selected the web-cam, the face detection is triggered to display the live video with the detection boundary around the face. One thing to keep in mind with video processing is that what we see with the human eye can be misleading. Not every frame in a video will render an image with a recognizable face. Some frames are transitional and it may take several frames for an image to come into full focus and for the features to be recognized. This is why some frames that we think should have a face don't have one. This is not always indicative of a failure in the recognition it is just due to the nature of videos. Figure 6.2 shows the results of the face detection on the video from web-cam which is also called the live video. As mentioned earlier, the connected web-cam is already configured into the JMStudio. Now, we click on the webcam button which shows the webcam video at the output. Again, if any face is present in the video then that face is displayed with the boundary around it. If no face is present then the live video will be displayed as is. 34 Chapter 7 FACE RECOGNITION ON VIDEO In this part of the project, we capture the video either from the video file or directly from the web-cam and recognize the particular person present in the video frames. The concept is similar to one used in the face recognition on still images. Here, we also store some sample images in the database. The location of this database is set in the system. When the video from any source runs in the face detection system then each and every frame of video stream is going to be compared with the images present in the database. If there is a match then the results will be displayed as a matched name. For the purpose of recognition, the signature of each and every frame of the video file is calculated. The signature of the stored image is also calculated. The distance of two images is calculated just as in the case of image face recognition. But here the threshold value is set up already. The threshold value is taken as 100. If the distance is less than equal to the threshold value than the result is the hit and the image stored in the database is identified as the match corresponding to the frame displaying at the output. If the signature is greater than the threshold value than the there is no match and no identification is revealed. The visual results of the recognition on the video of video file are displayed in the figure 7.1 and figure 7.2. To recognize, the frames of the video to be recognized are taken and stored in the folder and then the same folder is selected as a matching folder. 35 Figure 7.1: Video Face Recognition on person 1(Sam) After the locations of the video file and the corresponding database are selected then we check the box of “Search for Image”. After that we click on the “Video File” button. By doing that the video will be displayed from the selected video file and the face recognition system matches each and every frame of the video to the images stored in the database. Again the matched results are displayed with the name of the person to be identified. As we can see in figure 7.1 the person name is Sam whose picture is stored in the database. After matching the results are displayed with name “Sam” at the output. Similarly, the second person present in the video whose name is “Amy” is being 36 recognized as shown in the figure 7.2. Similar results can be displayed on other video files. Figure 7.2: Video Face Recognition on person 2(Amy) Apart from this, one more condition is tested when there is no picture present in the database of the person being displayed in the video file. This is shown in the figure 7.3. The recognition system will still look into the database to find a match of the person being present in the video. But, with no picture in database results in no matching results are displayed. 37 Figure 7.3: No matching results on video file’s video In the case of live video from a web-cam background noise must be minimized in order to recognize a face. To accomplish this first a signature is calculated for each image in the database. The signature is calculated by first performing facial recognition on the image. The signature is then calculated from the rectangular area that comprises the face in the image. This effectively removes any background interference. The same procedure is applied to each video frame. To determine a match we calculate the percentage difference between a video frame and each image in the database. The frame that has the closest percentage match and that is also above the threshold is 38 considered the matching face. The threshold in this case is set to a value between 0 and 100. Figure 7.4: Web-Cam recognition results The result on the live video from web-cam is displayed by figure 7.4. In this case the person to be identified has the images already stored in the database. This database is selected in recognition. Again we select the “Search for Image” button and click on the “WebCam” button. The live video from web-cam will start displaying at the output. The 39 recognition system searches the database against the person in the live video. If there is a match then the results are displayed at the output with name whose identity was being revealed. On the other hand if the detected person’s image is not stored in the database then no matching results will be displayed at the output. This scenario is explained by figure 7.5. Here the detection system is able to detect the person with the boundary around the face but recognition will display no matching results because the person is not present in the database. Figure 7.5: Live video face recognition with no matching results 40 Chapter 8 IMPLEMENTATION RESULTS To verify the simulation results, about 50 images of the people of different races are downloaded from websites. These images are of Asian, American and African people. The images are a mixture of men, women, people with glasses, children, people with beards and some non face images. The efficiency of the project is about 81%. The time required to perform the face detection on the single image is 0.32 Seconds. In the database of 36 images it takes 1.31 seconds to display the matching results for the image recognition part. The time project takes to detect the face from the video file is 1.27seconds. To detect the face from live video using web cam, the required time is 2.60 seconds. When recognizing the face present in the video file from the database of 25 pictures, it takes 3.35 seconds to show the matching results at output. Using the same database on the live video of the web-cam, the time to detect and match the results is 5.04 seconds. This time can be reduced on the smaller database and it can be more on the larger database. It is because the recognition compares each and every image of the database before returning the closest matching image. 41 Chapter 9 CONCLUSION The feature based face detection system is implemented with the provision of detection and the recognition on live video and the stored video. The image face detection is implemented first and then the same system is used to detect from video sources. The recognition system has also been implemented on the image and the video files. The accuracy of the system is achieved above 80%. The project is good at the pictures of the people of different races and colors. The most interesting and challenging parts are the live video face detection and the recognition. The project is good to detect the frontal faces present in the images and the video files but not able to detect the sideviews faces. The failure of detection on the pictures with very dark backgrounds colors are also the limitation of the system just like other systems. Overall it is a good project by which I have gained valuable knowledge of image processing and the steps required for any successful face detection. The advancement can be achieved as the future goal to make most parts of the project automated for surveillance and vision based applications. 42 BIBLIOGRAPHY 1. Erik Hjelmas, Boon Kee Low, "Face Detection: A Survey",Computer Vision and Image Understanding 83 . March 28, 2001, pages 236–274 . 2. Adriana Kovashka and Margaret Martonosi, “Future- Based Face Recognition for Identification of Criminals vs. Identification for Cashless Purchase”, January 12 2010, http://userweb.cs.utexas.edu/~adriana/kovashka_dmp_paper.pdf 3. S.A. Pardeshi and S.N. Talbar,”Local Feature Based Automatic Face Recognition System”, January 18, 2010, http://www.jiit.ac.in/jiit/ic3/IC3_2008/IC32008/ALG2_20.pdf 4. Quanni Zhang and Ebroul Izquierdo, “Multi-Feature Based Face Detection”, January 18, 2010, http://www.acemedia.org/aceMedia/files/document/vie06-qmul-3.pdf 5. Phuong-Trinh Pham-Ngoc, Quang-Linh Huynh, “Robust Face Detection under Challenges of Rotation, Pose and Occlusion”, January 26, 2010, http://www.fas.hcmut.edu.vn/webhn10/Baocao/PDF/YVSM_PhuongTrinh.pdf 6. D. Beymer,” Face Recognition under Varying Pose”, In IEEE Conf. on Comp. Vision and Pattern Recognition, 1994, pages 756-761. 7. “Image Processing Book”, February 2, 2010, http://www.imageprocessingbook.com/ 8. “Color Space”, February 4, 2010, http://en.wikipedia.org/wiki/Color_space 9. “The RGB Colorspace”, February 4, 2010, http://gimpsavvy.com/BOOK/index.html?node50.html 10. “Java (Programming Language)”, February 10, 2010, http://en.wikipedia.org/wiki/Java_%28programming_language%29 11. “Java Advanced Imaging”, February 10, 2010, http://en.wikipedia.org/wiki/Java_Advanced_Imaging 12. Sun Developer Network (SDN), “Java Advanced Imaging API”, February 10, 2010, http://java.sun.com/products/java-media/jai/iio.html 13. Java Media Framework,” Introduction”, February 10, 2010, http://grack.com/downloads/school/enel619.10/report/java_media_framework.html 14. “Java Development Kit”, February 10, 2010, http://en.wikipedia.org/wiki/java_development_kit 43 15. Dana Nourie, “Getting Started with the NetBeans IDE Tutorial”, February 12 2010 http://java.sun.com/developer/onlineTraining/tools/netbeans_part1/ 16. Ming-Hsuan Yang, “Face Detection”, February 26, 2010, http://faculty.ucmerced.edu/mhyang/papers/face-detection-chapter.pdf 17. Henry Chang and Ulises Robles, “Skin Segmentation”, November 10, 2009, http://www-cs-students.stanford.edu/~robles/ee368/skinsegment.html 18. Bill Green, “Edge Detection Tutorial”, November 18, 2009, http://www.pages.drexel.edu/~weg22/edge.html 19. A.N.Rajagopalan, K. Sandeep "Human Face Detection in Cluttered Color Images using Skin color and Edge Information" Indian Conference on Computer Vision, Graphics and Image Processing December 16-18, 2002. 20. Liya Ding and Aleix M. Martinez,”Precise Detailed Detection of Faces and Facial Features”, January 10, 2010, http://www.ece.osu.edu/~aleix/CVPR08.pdf 44 ATTACHMENTS Project Coding Classes: Edge.java: import javax.media.jai.JAI; import javax.media.jai.KernelJAI; import javax.media.jai.PlanarImage; import javax.swing.JFrame; /** * This class demonstrates the use of the convolve operator for edge-detection on an * image using the convolve operator and the Sobel horizontal kernel. */ public class Edge { // Create a constant array with the Sobel horizontal kernel. public Edge() { } public void EdgeDetect(String path) { float[] kernelMatrix = { -1, -2, -1, 45 0, 0, 0, 1, 2, 1 }; // Read the image. PlanarImage input = JAI.create("fileload", path); // Create the kernel using the array. KernelJAI kernel = new KernelJAI(3,3,kernelMatrix); // Run the convolve operator, creating the processed image. PlanarImage output = JAI.create("convolve", input, kernel); // Create a JFrame for displaying the results. JFrame frame = new JFrame("Sobel horizontal border of the image "+path); // Add to the JFrame's ContentPane an instance of // DisplayTwoSynchronizedImages, which will contain the original and // processed image. frame.getContentPane().add(new DisplayTwoSynchronizedImages(input,output)); // Set the closing operation so the application is finished. frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); frame.pack(); // Adjust the frame size using preferred dimensions. frame.setVisible(true); // Show the frame. } } FrameAcess.java: /* * To change this template, choose Tools | Templates 46 * and open the template in the editor. */ import java.awt.*; import javax.media.*; import javax.media.control.TrackControl; import javax.media.format.*; /** * Program to access individual video frames by using a * "pass-thru" codec. The codec is inserted into the data flow * path. As data passes through this codec, a callback is invoked * for each frame of video data. */ public class FrameAccess extends Frame implements ControllerListener { Processor p; Object waitSync = new Object(); boolean stateTransitionOK = true; MediaLocator media = null; public boolean open(MediaLocator ml, String search_folder) { try { p = Manager.createProcessor(ml); 47 } catch (Exception e) { System.err.println("Failed to create a processor from the given url: " + e); return false; } // set the search folder if (search_folder != null) { GlobalSearchFolder.search = true; GlobalSearchFolder.search_folder = search_folder; GlobalSearchFolder.temp_name = search_folder +"\\temp.jpg"; } else GlobalSearchFolder.search = false; media = ml; p.addControllerListener(this); // Put the Processor into configured state. p.configure(); if (!waitForState(p.Configured)) { System.err.println("Failed to configure the processor."); return false; } 48 // So I can use it as a player. p.setContentDescriptor(null); // Obtain the track controls. TrackControl tc[] = p.getTrackControls(); if (tc == null) { System.err.println("Failed to obtain track controls from the processor."); return false; } // Search for the track control for the video track. TrackControl videoTrack = null; for (int i = 0; i < tc.length; i++) { if (tc[i].getFormat() instanceof VideoFormat) { videoTrack = tc[i]; break; } } if (videoTrack == null) { System.err.println("The input media does not contain a video track."); return false; } 49 System.err.println("Video format: " + videoTrack.getFormat()); // Instantiate and set the frame access codec to the data flow path. try { Codec codec[] = { new PreAccessCodec(), new PostAccessCodec()}; videoTrack.setCodecChain(codec); } catch (UnsupportedPlugInException e) { System.err.println("The process does not support effects."); } // Realize the processor. p.prefetch(); if (!waitForState(p.Prefetched)) { System.err.println("Failed to realize the processor."); return false; } // Display the visual & control component if there's one. setLayout(new BorderLayout()); Component cc; 50 Component vc; if ((vc = p.getVisualComponent()) != null) { add("Center", vc); } if ((cc = p.getControlPanelComponent()) != null) { add("South", cc); } // Start the processor. p.start(); setVisible(true); return true; } public void addNotify() { super.addNotify(); pack(); } /** * Block until the processor has transitioned to the given state. * Return false if the transition failed. 51 */ boolean waitForState(int state) { synchronized (waitSync) { try { while (p.getState() != state && stateTransitionOK) waitSync.wait(); } catch (Exception e) {} } return stateTransitionOK; } public void controllerUpdate(ControllerEvent evt) { if (evt instanceof ConfigureCompleteEvent || evt instanceof RealizeCompleteEvent || evt instanceof PrefetchCompleteEvent) { synchronized (waitSync) { stateTransitionOK = true; waitSync.notifyAll(); } } else if (evt instanceof ResourceUnavailableEvent) { synchronized (waitSync) { stateTransitionOK = false; waitSync.notifyAll(); 52 } } else if (evt instanceof EndOfMediaEvent) { p.close(); } } } DisplayTwoSynchronizedImages.java: import java.awt.GridLayout; import java.awt.event.AdjustmentEvent; import java.awt.event.AdjustmentListener; import java.awt.image.RenderedImage; import javax.swing.JPanel; import javax.swing.JScrollPane; import com.sun.media.jai.widget.DisplayJAI; /** * This class represents a JPanel which contains two scrollable instances of * DisplayJAI. The scrolling bars of both images are synchronized so scrolling one * image will automatically scroll the other. */ 53 public class DisplayTwoSynchronizedImages extends JPanel implements AdjustmentListener { /** The DisplayJAI for the first image. */ protected DisplayJAI dj1; /** The DisplayJAI for the second image. */ protected DisplayJAI dj2; /** The JScrollPane which will contain the first of the images */ protected JScrollPane jsp1; /** The JScrollPane which will contain the second of the images */ protected JScrollPane jsp2; /** * Creates an instance of this class, setting the components' layout, creating * two instances of DisplayJAI for the two images and creating/registering * event handlers for the scroll bars. * * @param im1 the first image (left side) * @param im2 the second image (right side) */ public DisplayTwoSynchronizedImages(RenderedImage im1, RenderedImage im2) { super(); setLayout(new GridLayout(1, 2)); dj1 = new DisplayJAI(im1); // Instances of DisplayJAI for the 54 dj2 = new DisplayJAI(im2); // two images jsp1 = new JScrollPane(dj1); // JScrollPanes for the both jsp2 = new JScrollPane(dj2); // instances of DisplayJAI add(jsp1); add(jsp2); // Retrieve the scroll bars of the images and registers adjustment // listeners to them. // Horizontal scroll bar of the first image. jsp1.getHorizontalScrollBar().addAdjustmentListener(this); // Vertical scroll bar of the first image. jsp1.getVerticalScrollBar().addAdjustmentListener(this); // Horizontal scroll bar of the second image. jsp2.getHorizontalScrollBar().addAdjustmentListener(this); // Vertical scroll bar of the second image. jsp2.getVerticalScrollBar().addAdjustmentListener(this); } /** * This method changes the first image to be displayed. * @param newImage the new first image. */ public void setImage1(RenderedImage newimage) { dj1.set(newimage); repaint(); 55 } /** * This method changes the second image to be displayed. * @param newImage the new second image. */ public void setImage2(RenderedImage newimage) { dj2.set(newimage); repaint(); } /** * This method returns the first image. * @return the first image. */ public RenderedImage getImage1() { return dj1.getSource(); } /** * This method returns the second image. * @return the second image. */ 56 public RenderedImage getImage2() { return dj2.getSource(); } /** * This method will be called when any of the scroll bars of the instances of * DisplayJAI are changed. The method will adjust the scroll bar * of the other DisplayJAI as needed. * * @param e the AdjustmentEvent that occurred (meaning that one of the scroll * bars position has changed. */ public void adjustmentValueChanged(AdjustmentEvent e) { // If the horizontal bar of the first image was changed... if (e.getSource() == jsp1.getHorizontalScrollBar()) { // We change the position of the horizontal bar of the second image. jsp2.getHorizontalScrollBar().setValue(e.getValue()); } // If the vertical bar of the first image was changed... if (e.getSource() == jsp1.getVerticalScrollBar()) { // We change the position of the vertical bar of the second image. 57 jsp2.getVerticalScrollBar().setValue(e.getValue()); } // If the horizontal bar of the second image was changed... if (e.getSource() == jsp2.getHorizontalScrollBar()) { // We change the position of the horizontal bar of the first image. jsp1.getHorizontalScrollBar().setValue(e.getValue()); } // If the vertical bar of the second image was changed... if (e.getSource() == jsp2.getVerticalScrollBar()) { // We change the position of the vertical bar of the first image. jsp1.getVerticalScrollBar().setValue(e.getValue()); } } // end adjustmentValueChanged } EyeDetect.java: import java.net.URL; import java.awt.Component; import java.awt.Graphics; import java.awt.image.*; import java.awt.Color; import java.io.File; 58 import javax.imageio.*; import javax.swing.*; public class EyeDetect extends ImageIcon { static fdlibjni fdlib; static int n; static JFrame jf; static JPanel panel; static int w, h, threshold; static byte[] img; static StringBuilder sb = new StringBuilder(); public EyeDetect(URL url) { super(url); } static void receiveImage(String filepath) { jf = new JFrame(); panel = new JPanel(); fdlib = new fdlibjni(); threshold = 0; try { 59 System.out.println("call"); BufferedImage bi = ImageIO.read(new File(filepath)); w = bi.getWidth(); h = bi.getHeight(); System.out.println("image is " + w + "x" + h + " pixels"); img = new byte[w*h]; for (int y=0, i=0; y<h; y++) { for (int x=0; x<w; x++, i++) { //GET RGB VALUE TO FIND PIXEL COLOR int c = bi.getRGB(x,y); int r, g, b; //FIND RED COLOR r = (c&0x00ff0000)>>16; //FIND GREEN g = (c&0x0000ff00)>>8; //BLUE COLOR b = (c&0x000000ff); //CONVERT INTO GRAYSCALE COLOR img[i] = (byte) (.3*r + .59*g + .11*b); } } 60 System.out.println("detecting with threshold " + threshold); fdlib.detectfaces(img, w, h, threshold); n = fdlib.getndetections(); if (n==1) System.out.println("1 face found."); else System.out.println(n + " faces found."); java.io.File f = new java.io.File(filepath); EyeDetect pi = new EyeDetect(f.toURL()); JLabel l = new JLabel(pi); String infoImg ="File Name: "+f.getName() +"\nFile Path: "+f.getPath() +"\nTotal Numer Of Faces Found: "+n; System.out.println(infoImg); HomeFile.imageInformation(infoImg); panel.add(l); jf.getContentPane().add(panel); jf.pack(); jf.setTitle("Face Detection Interface"); jf.setVisible(true); } catch(Exception e) { 61 System.out.println(e); } } public EyeDetect() { } public void paintIcon(Component c, Graphics g, int x, int y) { super.paintIcon(c,g,x,y); for (int i=0; i<n; i++) { System.out.println("x:" + fdlib.getdetectionx(i) + " y:" + fdlib.getdetectiony(i) + " size:" + fdlib.getdetectionwidth(i)); g.setColor(Color.WHITE); //DRAW RECTANGLE TO IDENTIFY EYES g.drawRect((fdlib.getdetectionx(i)-(fdlib.getdetectionwidth(i)/8)20),(fdlib.getdetectiony(i)-(fdlib.getdetectionwidth(i)/8)20),fdlib.getdetectionwidth(i)/8,fdlib.getdetectionwidth(i)/8); g.drawRect((fdlib.getdetectionx(i)+(fdlib.getdetectionwidth(i)/8)+10),(fdlib.getdetectiony (i)-(fdlib.getdetectionwidth(i)/8)20),fdlib.getdetectionwidth(i)/8,fdlib.getdetectionwidth(i)/8); } System.out.println("paintIcon"); 62 } } GlobalSearchFolder.java: import javax.media.*; public class GlobalSearchFolder { private GlobalSearchFolder() {} public static String search_folder, temp_name; public static boolean search; public static double threshold; public static Processor p; public static int x, y, width, height; public static boolean webcam; } HomeFile.java: import java.awt.*; import java.awt.event.*; import java.io.*; import javax.swing.*; import javax.swing.border.*; 63 public class HomeFile implements ActionListener { JFrame jf; JPanel mainPanel,imageChoose,videoChoose,dispInfo,dispImage,imageInfo,imageDispInfo; JLabel banner,videoBanner,imageView; JCheckBox search_image; ImageIcon icon; JTextField filePath, folderPath; static JTextArea aboutImage,aboutImageInfo; JButton selectFile_but,chooseFile_But,videoProcess,faceDetectBut,imageComapre,skinDetect,ey eDetect,noseDetect,mouthDetect,edgeDetect; JButton webcamProcess, chooseFolder, detectAndSearch; JFileChooser openDialog1, folderDialog; private JScrollPane scrollPane,scrollPaneInfo; public static final String BUTTON_TEXT = "Select Image/Video"; Container content; SearchImages search_images; public HomeFile() { //CREATE JFRAME WITH TITLE jf = new JFrame("Face Detection"); Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize(); 64 try { UIManager.setLookAndFeel(UIManager.getSystemLookAndFeelClassName()); } catch (ClassNotFoundException ex) { ex.printStackTrace(); } catch (IllegalAccessException ex) { ex.printStackTrace(); } catch (UnsupportedLookAndFeelException ex) { ex.printStackTrace(); } catch (InstantiationException ex) { ex.printStackTrace(); } // Create Main Pannel mainPanel = new JPanel(); mainPanel.setLayout(null); mainPanel.setSize(500,500); mainPanel.setBackground(new Color(139, 125, 107 )); String str = "<html><hr color=\"#FFFFFF\"><p align=\"center\" color=\"#292421\" style=\"padding:10px 100px 10px 10px;text-align:\"center\" bgcolor=\"#e2e2c6\" >" +"&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs p;&nbsp;&nbsp;" +"&nbsp;&nbsp;" +"Image Face Detection" 65 +"</p><hr></html>"; //CREATE LABEL banner = new JLabel(str); banner.setBounds(120,10,screenSize.width,100); banner.setFont(new Font("tahoma, arial, sans-serif", Font.BOLD, 30)); banner.setForeground(new Color(162,184,73)); mainPanel.add(banner); //CREATE IMAGE PANEL imageChoose = new JPanel(); TitledBorder titleCom = BorderFactory.createTitledBorder("Choose JPEG Image Files"); titleCom.setTitleColor(new Color(41, 36, 33 )); titleCom.setTitleFont(new Font("tahoma, arial, sans-serif", Font.BOLD, 12)); imageChoose.setBorder(titleCom); imageChoose.setLayout(null); imageChoose.setBounds(20,125,700,100); imageChoose.setBackground(new Color(205, 186, 150 )); //CREATE TEXT FIELD FOR PATH filePath = new JTextField(); filePath.setBounds(80,25,350,20); filePath.setBorder(BorderFactory.createBevelBorder(BevelBorder.LOWERED)); imageChoose.add(filePath); 66 //CREATE FILE CHOOSER chooseFile_But = new JButton("Browse File"); chooseFile_But.setBounds(450,25,100,20); chooseFile_But.addActionListener(this); imageChoose.add(chooseFile_But); // create text box image folder folderPath = new JTextField(); folderPath.setBounds(80, 60, 350, 20); folderPath.setBorder(BorderFactory.createBevelBorder(BevelBorder.LOWERED)); imageChoose.add(folderPath); // create folder chooser chooseFolder = new JButton("Choose Search Folder"); chooseFolder.setBounds(450, 60, 150, 20); chooseFolder.addActionListener(this); imageChoose.add(chooseFolder); mainPanel.add(imageChoose); //CREATE VIDEO CHOOSE PANEL videoChoose= new JPanel(); 67 TitledBorder videoCom = BorderFactory.createTitledBorder("Video Face Detection"); videoCom.setTitleColor(new Color(41, 36, 33 )); videoCom.setTitleFont(new Font("tahoma, arial, sans-serif", Font.BOLD, 12)); videoChoose.setBorder(videoCom); videoChoose.setLayout(null); videoChoose.setBounds(730,125,250,100); videoChoose.setBackground(new Color(205, 186, 150 )); String strVideo = "<html><p>" +"*&nbsp;From<br>"; videoBanner = new JLabel(strVideo); videoBanner.setBounds(10,5,screenSize.width,100); videoBanner.setFont(new Font("tahoma, arial, sans-serif", Font.BOLD, 10)); videoBanner.setForeground(Color.DARK_GRAY); videoProcess = new JButton("Video File"); videoProcess.setBounds(120,70,120,20); videoProcess.addActionListener(this); webcamProcess = new JButton("WebCam"); webcamProcess.setBounds(120,45,120,20); webcamProcess.addActionListener(this); 68 search_image = new JCheckBox("Search for Image", false); search_image.setBounds(120, 20, 120, 20); search_image.addActionListener(this); //ADD COMPONENTS videoChoose.add(videoProcess); videoChoose.add(webcamProcess); videoChoose.add(videoBanner); videoChoose.add(search_image); mainPanel.add(videoChoose); //CREATE IMAGE PANEL dispImage= new JPanel(); TitledBorder dispImageBorder = BorderFactory.createTitledBorder("Image Before Process"); dispImageBorder.setTitleColor(new Color(41, 36, 33 )); dispImage.setBorder(dispImageBorder); dispImage.setLayout(null); dispImage.setBounds(20,230,700,340); dispImage.setBackground(new Color(205, 186, 150 )); 69 faceDetectBut = new JButton("Detect Face"); faceDetectBut.setBounds(550,12,140,20); faceDetectBut.setVisible(false); faceDetectBut.addActionListener(this); dispImage.add(faceDetectBut); detectAndSearch = new JButton("Detect and Search"); detectAndSearch.setBounds(550, 40, 140, 20); detectAndSearch.setVisible(false); detectAndSearch.addActionListener(this); dispImage.add(detectAndSearch); skinDetect = new JButton("Skin Color Detection"); skinDetect.setBounds(550,42,140,20); skinDetect.setVisible(false); skinDetect.addActionListener(this); dispImage.add(skinDetect); eyeDetect = new JButton("Eye Detection"); eyeDetect.setBounds(550,72,140,20); eyeDetect.setVisible(false); eyeDetect.addActionListener(this); dispImage.add(eyeDetect); 70 noseDetect = new JButton("Nose Detection"); noseDetect.setBounds(550,102,140,20); noseDetect.setVisible(false); noseDetect.addActionListener(this); dispImage.add(noseDetect); mouthDetect = new JButton("Mouth Detection"); mouthDetect.setBounds(550,132,140,20); mouthDetect.setVisible(false); mouthDetect.addActionListener(this); dispImage.add(mouthDetect); edgeDetect = new JButton("Edge Detection"); edgeDetect.setBounds(550,162,140,20); edgeDetect.setVisible(false); edgeDetect.addActionListener(this); dispImage.add(edgeDetect); imageView = new JLabel(); imageView.setBounds(10,5,690,330); dispImage.add(imageView); mainPanel.add(dispImage); imageInfo= new JPanel(); 71 TitledBorder dispImgInfoBorder = BorderFactory.createTitledBorder("Image Info"); dispImgInfoBorder.setTitleColor(new Color(41, 36, 33 )); imageInfo.setBorder(dispImgInfoBorder); imageInfo.setLayout(null); imageInfo.setBounds(730,230,250,340); imageInfo.setBackground(new Color(205, 186, 150 )); aboutImage = new JTextArea(); aboutImage.setEditable(false); aboutImage.setBackground(new Color(192,192,192)); scrollPane = new JScrollPane(aboutImage); scrollPane.setBounds(5,15,240,320); imageInfo.add(scrollPane); imageDispInfo= new JPanel(); TitledBorder dispImageInfo = BorderFactory.createTitledBorder("Details Of the Image"); dispImageBorder.setTitleColor(new Color(41, 36, 33 )); imageDispInfo.setBorder(dispImageBorder); imageDispInfo.setLayout(null); imageDispInfo.setBounds(20,575,960,130); imageDispInfo.setBackground(new Color(205, 186, 150 )); aboutImageInfo = new JTextArea(); 72 aboutImageInfo.setEditable(false); aboutImageInfo.setBackground(new Color(192,192,192)); scrollPaneInfo = new JScrollPane(aboutImageInfo); scrollPaneInfo.setBounds(5,15,950,110); imageDispInfo.add(scrollPaneInfo); mainPanel.add(imageDispInfo); mainPanel.add(imageInfo); //SET SIZE ,VISIBILITY AND WINDOW CLOSE OPTION jf.add(mainPanel); jf.setSize(screenSize.width,screenSize.height); jf.setVisible(true); jf.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE); } //ACTION PERFORMED FOR CLICK EVENT public void actionPerformed(ActionEvent ae) { //OPEN DIALOG WINDOW if(ae.getSource()==chooseFile_But) { openDialog1 = new JFileChooser(); openDialog1.setFileSelectionMode(JFileChooser.FILES_ONLY); openDialog1.setDialogTitle("Choose JPEG Format File"); 73 openDialog1.showDialog(content, BUTTON_TEXT); File selectedFiles1 = openDialog1.getSelectedFile(); filePath.setText(selectedFiles1.toString()); filePath.setEditable(false); icon = new ImageIcon(selectedFiles1.toString()); //VISIBLE BUTTONS imageView.setIcon(icon); faceDetectBut.setVisible(true); skinDetect.setVisible(false); eyeDetect.setVisible(false); noseDetect.setVisible(false); mouthDetect.setVisible(false); edgeDetect.setVisible(false); } else if (ae.getSource() == chooseFolder) { folderDialog = new JFileChooser(); folderDialog.setFileSelectionMode(JFileChooser.DIRECTORIES_ONLY); folderDialog.setDialogTitle("Choose the Image Folder"); 74 folderDialog.showDialog(content, "Choose Folder"); File selectedFolder = folderDialog.getSelectedFile(); folderPath.setText(selectedFolder.toString()); folderPath.setEditable(false); detectAndSearch.setVisible(true); } else if(ae.getSource()==videoProcess) { // detect faces in a video if (search_image.isSelected() == true) { // detect and search fdVideoTest.receiveVideo(filePath.getText(), folderPath.getText()); } else { // just detect fdVideoTest.receiveVideo(filePath.getText(), null); } } 75 else if(ae.getSource()==webcamProcess) { // detect faces in a webcam if (search_image.isSelected() == true) { // detect and search fdVideoTest.receiveWebCam(folderPath.getText()); } else { // just detect fdVideoTest.receiveWebCam(null); } } else if(ae.getSource()==faceDetectBut) { //CALL FDTEST CLASS FOR DETECT FACE fdtest.receiveImage(filePath.getText(), null); aboutImageInfo.setText(ShowImageInfo.printInfo(filePath.getText())); } else if (ae.getSource() == detectAndSearch) { // detect a face and search 76 fdtest.receiveImage(filePath.getText(), folderPath.getText()); aboutImageInfo.setText(ShowImageInfo.printInfo(filePath.getText())); } else if(ae.getSource()==skinDetect) { aboutImageInfo.setText(fdtest.sb.toString()); } else if(ae.getSource()==eyeDetect) { EyeDetect.receiveImage(filePath.getText()); } else if(ae.getSource()==noseDetect) { NoseDetection.receiveImage(filePath.getText()); } else if(ae.getSource()==mouthDetect) { MouthDetection.receiveImage(filePath.getText()); 77 } else if(ae.getSource()==edgeDetect) { new Edge().EdgeDetect(filePath.getText()); } } public static void main(String args[]) { new HomeFile(); } static void imageInformation(String info) { //SET IMAGE INFORMATION System.out.println("call imageinfo"); aboutImage.append(info); } } JPEGImageFileFilter.java: 78 import java.io.File; import javax.swing.filechooser.FileFilter; /* * This class implements a generic file name filter that allows the listing/selection * of JPEG files. */ public class JPEGImageFileFilter extends FileFilter implements java.io.FileFilter { public boolean accept(File f) { if (f.getName().toLowerCase().endsWith(".jpeg")) return true; if (f.getName().toLowerCase().endsWith(".jpg")) return true; return false; } public String getDescription() { return "JPEG files"; } } MouthDetection.java: 79 import java.net.URL; import java.awt.Component; import java.awt.Graphics; import java.awt.image.*; import java.awt.Color; import java.io.File; import javax.imageio.*; import javax.swing.*; public class MouthDetection extends ImageIcon { static fdlibjni fdlib; static int n; static JFrame jf; static JPanel panel; static int w, h, threshold; static byte[] img; static StringBuilder sb = new StringBuilder(); public MouthDetection(URL url) { super(url); } static void receiveImage(String filepath) { 80 jf = new JFrame(); panel = new JPanel(); fdlib = new fdlibjni(); threshold = 0; try { System.out.println("call"); BufferedImage bi = ImageIO.read(new File(filepath)); w = bi.getWidth(); h = bi.getHeight(); System.out.println("image is " + w + "x" + h + " pixels"); img = new byte[w*h]; for (int y=0, i=0; y<h; y++) { for (int x=0; x<w; x++, i++) { int c = bi.getRGB(x,y); int r, g, b; r = (c&0x00ff0000)>>16; g = (c&0x0000ff00)>>8; b = (c&0x000000ff); img[i] = (byte) (.3*r + .59*g + .11*b); } } 81 System.out.println("detecting with threshold " + threshold); fdlib.detectfaces(img, w, h, threshold); n = fdlib.getndetections(); if (n==1) System.out.println("1 face found."); else System.out.println(n + " faces found."); java.io.File f = new java.io.File(filepath); MouthDetection pi = new MouthDetection(f.toURL()); JLabel l = new JLabel(pi); String infoImg ="File Name: "+f.getName() +"\nFile Path: "+f.getPath() +"\nTotal Numer Of Faces Found: "+n; System.out.println(infoImg); HomeFile.imageInformation(infoImg); panel.add(l); jf.getContentPane().add(panel); jf.pack(); jf.setTitle("Face Detection Interface"); jf.setVisible(true); } catch(Exception e) { System.out.println(e); 82 } } public MouthDetection() { } public void paintIcon(Component c, Graphics g, int x, int y) { super.paintIcon(c,g,x,y); for (int i=0; i<n; i++) { System.out.println("x:" + fdlib.getdetectionx(i) + " y:" + fdlib.getdetectiony(i) + " size:" + fdlib.getdetectionwidth(i)); g.setColor(Color.WHITE); //DRAW RECTANGLE TO IDENTIFY MOUTH g.drawRect((fdlib.getdetectionx(i)(fdlib.getdetectionwidth(i)/8)),(fdlib.getdetectiony(i)+(fdlib.getdetectionwidth(i)/4)),fdlib .getdetectionwidth(i)/4,fdlib.getdetectionwidth(i)/8); } System.out.println("paintIcon"); } } 83 NoseDetection.java: import java.net.URL; import java.awt.Component; import java.awt.Graphics; import java.awt.image.*; import java.awt.Color; import java.io.File; import javax.imageio.*; import javax.swing.*; public class NoseDetection extends ImageIcon { static fdlibjni fdlib; static int n; static JFrame jf; static JPanel panel; static int w, h, threshold; static byte[] img; static StringBuilder sb = new StringBuilder(); public NoseDetection(URL url) { super(url); } 84 static void receiveImage(String filepath) { jf = new JFrame(); panel = new JPanel(); fdlib = new fdlibjni(); threshold = 0; try { System.out.println("call"); BufferedImage bi = ImageIO.read(new File(filepath)); w = bi.getWidth(); h = bi.getHeight(); System.out.println("image is " + w + "x" + h + " pixels"); img = new byte[w*h]; for (int y=0, i=0; y<h; y++) { for (int x=0; x<w; x++, i++) { int c = bi.getRGB(x,y); int r, g, b; r = (c&0x00ff0000)>>16; g = (c&0x0000ff00)>>8; b = (c&0x000000ff); img[i] = (byte) (.3*r + .59*g + .11*b); } 85 } System.out.println("detecting with threshold " + threshold); fdlib.detectfaces(img, w, h, threshold); n = fdlib.getndetections(); if (n==1) System.out.println("1 face found."); else System.out.println(n + " faces found."); java.io.File f = new java.io.File(filepath); NoseDetection pi = new NoseDetection(f.toURL()); JLabel l = new JLabel(pi); String infoImg ="File Name: "+f.getName() +"\nFile Path: "+f.getPath() +"\nTotal Numer Of Faces Found: "+n; System.out.println(infoImg); HomeFile.imageInformation(infoImg); panel.add(l); jf.getContentPane().add(panel); jf.pack(); jf.setTitle("Face Detection Interface"); jf.setVisible(true); } catch(Exception e) 86 { System.out.println(e); } } public NoseDetection() { } public void paintIcon(Component c, Graphics g, int x, int y) { super.paintIcon(c,g,x,y); for (int i=0; i<n; i++) { System.out.println("x:" + fdlib.getdetectionx(i) + " y:" + fdlib.getdetectiony(i) + " size:" + fdlib.getdetectionwidth(i)); g.setColor(Color.WHITE); //DRAW RECTANGLE TO IDENTIFY NOSE g.drawRect((fdlib.getdetectionx(i)(fdlib.getdetectionwidth(i)/8)),(fdlib.getdetectiony(i)),fdlib.getdetectionwidth(i)/4,fdlib.g etdetectionwidth(i)/8); } System.out.println("paintIcon"); } 87 } PostAccessCodec.java: import javax.media.*; import javax.media.Format; import javax.media.format.*; /** * * @author Jalwinder */ public class PostAccessCodec extends PreAccessCodec { // We'll advertize as supporting all video formats. public PostAccessCodec() { supportedIns = new Format [] { new RGBFormat() }; } 88 /** * Callback to access individual video frames. */ void accessFrame(Buffer frame) { // do something } public String getName() { return "Post-Access Codec"; } } PreAccessCodec.java: import java.awt.*; import javax.media.*; import javax.media.Format; import javax.media.format.*; import javax.media.util.BufferToImage; import java.awt.image.BufferedImage; import javax.media.util.ImageToBuffer; import java.io.File; import javax.imageio.ImageIO; 89 /********************************************************* * Inner class. * * A pass-through codec to access to individual frames. *********************************************************/ public class PreAccessCodec implements Codec { static int frame_number = 0; SearchImages search_images = null; static boolean skip_frame = false; /** * The code for a pass through codec. */ // We'll advertize as supporting all video formats. protected Format supportedIns[] = new Format [] { new VideoFormat(null) }; 90 // We'll advertize as supporting all video formats. protected Format supportedOuts[] = new Format [] { new VideoFormat(null) }; Format input = null, output = null; public String getName() { return "Pre-Access Codec"; } // No op. public void open() { } // No op. public void close() { } // No op. public void reset() { } public Format [] getSupportedInputFormats() { return supportedIns; 91 } public Format [] getSupportedOutputFormats(Format in) { if (in == null) return supportedOuts; else { // If an input format is given, we use that input format // as the output since we are not modifying the bit stream // at all. Format outs[] = new Format[1]; outs[0] = in; return outs; } } public Format setInputFormat(Format format) { input = format; return input; } public Format setOutputFormat(Format format) { output = format; return output; } 92 public int process(Buffer in, Buffer out) { // This is the "Callback" to access individual frames. // in is the incoming frame, out is the outgoing frame. BufferedImage bi = null; int w = 0, h = 0, n; byte[] image_buffer = null; fdlibjni fdlib; File temp_file = null; String found_image = null; String infoImg; int this_x, this_y, this_width, this_height; if (GlobalSearchFolder.search == true) temp_file = new File(GlobalSearchFolder.temp_name); fdlib = new fdlibjni(); // convert the input buffer to an Image BufferToImage stopBuffer = new BufferToImage((VideoFormat) in.getFormat()); Image img = stopBuffer.createImage(in); if (img != null && skip_frame == false) 93 { frame_number++; // convert the img to a BufferedImage bi = new BufferedImage(img.getWidth(null), img.getHeight(null), BufferedImage.TYPE_INT_RGB); // to access the raster Graphics2D g2 = bi.createGraphics(); g2.drawImage(img, 0, 0, null); img.flush(); w = bi.getWidth(); h = bi.getHeight(); System.out.println("image is " + w + "x" + h + " pixels"); image_buffer = new byte[w * h]; for (int y = 0, i = 0; y < h; y++) { for (int x = 0; x < w; x++, i++) { //GET RGB VALUE TO FIND PIXEL COLOR int c = bi.getRGB(x,y); 94 int r, g, b; //FIND RED COLOR r = (c&0x00ff0000)>>16; //FIND GREEN g = (c&0x0000ff00)>>8; //BLUE COLOR b = (c&0x000000ff); //CONVERT INTO GRAYSCALE COLOR image_buffer[i] = (byte) (.3*r + .59*g + .11*b); } } // detect fdlib.detectfaces(image_buffer, w, h, 0); //GET THE COLOR VALUE AND DETECT IMAGE n = fdlib.getndetections(); infoImg = "\nFrame Number = " + frame_number + "\nNumber of Faces Found = " +n ; if (n > 0) { // search for a similar image if (GlobalSearchFolder.search == true) { 95 if (search_images == null) { search_images = new SearchImages(); if (GlobalSearchFolder.webcam == false) search_images.buildImageList(GlobalSearchFolder.search_folder); else search_images.BuildBufferedImageList(GlobalSearchFolder.search_folder); } // save the file try { ImageIO.write(bi, "jpg", temp_file); this_x = fdlib.getdetectionx(0); this_y = fdlib.getdetectiony(0); this_width = fdlib.getdetectionwidth(0); if (GlobalSearchFolder.webcam == true) found_image = search_images.GetMatch(bi, this_x, this_y, this_width, this_width); else found_image = search_images.findImage(GlobalSearchFolder.temp_name); temp_file.delete(); } 96 catch (Exception e) { found_image = null; } } } for (int i = 0; i < n; i++) { g2.setColor(Color.WHITE); //DRAW RECTANGLE TO IDENTIFY FACE g2.drawRect(fdlib.getdetectionx(i)-fdlib.getdetectionwidth(i), fdlib.getdetectiony(i)-fdlib.getdetectionwidth(i), fdlib.getdetectionwidth(i)*2,fdlib.getdetectionwidth(i)*2); g2.drawString(String.valueOf(i+1), fdlib.getdetectionx(i)(fdlib.getdetectionwidth(i)-10), fdlib.getdetectiony(i)-(fdlib.getdetectionwidth(i)-10)); } // replace the input buffer with the altered image if (n > 0) { in = ImageToBuffer.createBuffer(bi, 20); } 97 g2.dispose(); if (found_image != null) { int index_1, index_2; index_1 = found_image.lastIndexOf('\\'); index_2 = found_image.lastIndexOf('.'); String this_name = found_image.substring(index_1 + 1, index_2); infoImg = "\nFrame Number = " + frame_number + "\nNumber of Faces Found = " + n + "\n" + this_name + " was identified as a match."; } HomeFile.imageInformation(infoImg); skip_frame = true; } else skip_frame = false; // Swap the data between the input & output. Object data = in.getData(); in.setData(out.getData()); out.setData(data); 98 // Copy the input attributes to the output out.setFormat(in.getFormat()); out.setLength(in.getLength()); out.setOffset(in.getOffset()); return BUFFER_PROCESSED_OK; } public Object[] getControls() { return new Object[0]; } public Object getControl(String type) { return null; } } SearchImages.java: // class to find an image within an image directory that has a // close match to the input image import java.io.File; import java.awt.Color; 99 import java.awt.image.RenderedImage; import java.awt.image.renderable.ParameterBlock; import javax.media.jai.InterpolationNearest; import javax.media.jai.JAI; import javax.imageio.ImageIO; import javax.media.jai.iterator.RandomIter; import javax.media.jai.iterator.RandomIterFactory; import java.awt.Window; import java.awt.image.*; import java.io.FileOutputStream; import javax.media.jai.Interpolation; import java.awt.*; import java.io.FileInputStream; import java.awt.image.*; public class SearchImages { File[] image_files; // list of image files in the folder RenderedImage[] rothers; BufferedImage[] bothers; double [] bsignatures; static private Color[][] signature; private static final int baseSize = 300; static fdlibjni fdlib; BufferedImage bi; 100 int this_x, this_y, this_width, this_height; public int getFaces(File filepath) { int w, h, n; byte[] img; try { bi = ImageIO.read(filepath); } catch (Exception e) { return(0); } w = bi.getWidth(); h = bi.getHeight(); fdlib = new fdlibjni(); System.out.println("image is " + w + "x" + h + " pixels"); img = new byte[w*h]; for (int y=0, i=0; y<h; y++) { 101 for (int x=0; x<w; x++, i++) { //GET RGB VALUE TO FIND PIXEL COLOR int c = bi.getRGB(x,y); int r, g, b; //FIND RED COLOR r = (c&0x00ff0000)>>16; //FIND GREEN g = (c&0x0000ff00)>>8; //BLUE COLOR b = (c&0x000000ff); //CONVERT INTO GRAYSCALE COLOR img[i] = (byte) (.3*r + .59*g + .11*b); } } fdlib.detectfaces(img, w, h, 0); //GET THE COLOR VALUE AND DETECT IMAGE n = fdlib.getndetections(); if (n > 0) { this_x = fdlib.getdetectionx(0); this_y = fdlib.getdetectiony(0); this_width = fdlib.getdetectionwidth(0); this_height = fdlib.getdetectionwidth(0); 102 } return(n); } public int getFaces(BufferedImage b_image) { int w, h, n; byte[] img; w = b_image.getWidth(); h = b_image.getHeight(); fdlib = new fdlibjni(); img = new byte[w*h]; for (int y=0, i=0; y<h; y++) { for (int x=0; x<w; x++, i++) { //GET RGB VALUE TO FIND PIXEL COLOR int c = b_image.getRGB(x,y); int r, g, b; //FIND RED COLOR r = (c&0x00ff0000)>>16; 103 //FIND GREEN g = (c&0x0000ff00)>>8; //BLUE COLOR b = (c&0x000000ff); //CONVERT INTO GRAYSCALE COLOR img[i] = (byte) (.3*r + .59*g + .11*b); } } fdlib.detectfaces(img, w, h, 0); //GET THE COLOR VALUE AND DETECT IMAGE n = fdlib.getndetections(); if (n > 0) { this_x = fdlib.getdetectionx(0); this_y = fdlib.getdetectiony(0); this_width = fdlib.getdetectionwidth(0); this_height = fdlib.getdetectionwidth(0); } return(n); } public void BuildBufferedImageList(String path) { int i; 104 BufferedImage this_image; Image temp_image, image; int c_x, c_y, c_height; File dir = new File(path); image_files = dir.listFiles(new JPEGImageFileFilter()); bothers = new BufferedImage[image_files.length]; bsignatures = new double[image_files.length]; for (i = 0; i < image_files.length; i++) { try { this_image = ImageIO.read(image_files[i]); // get the face if ((getFaces(this_image)) == 1) { c_x = this_x - (this_width / 2); c_y = this_y - (this_width / 2); c_height = this_y - this_width; bothers[i] = this_image.getSubimage(c_x, c_y, this_width, c_height); bsignatures[i] = GetSignature(bothers[i]); 105 } else { bothers[i] = this_image; bsignatures[i] = GetSignature(bothers[i]); } } catch (Exception e) { bothers[i] = null; bsignatures[i] = -1; } } } public String GetMatch(BufferedImage match_image, int this_x, int this_y, int this_width, int this_height) { double match_sig, this_dist, small_dist, percent; BufferedImage crop_image; int index, i, c_x, c_y, c_height; index = -1; small_dist = 0; 106 c_x = this_x - (this_width / 2); c_y = this_y - (this_width / 2); c_height = this_y - this_width; crop_image = match_image.getSubimage(c_x, c_y, this_width, c_height); FileOutputStream os; try { os = new FileOutputStream("c:\\crap.png"); ImageIO.write(crop_image, "PNG", os); } catch (Exception e) { } match_sig = GetSignature(crop_image); for (i = 0; i < image_files.length; i++) { if (bsignatures[i] != -1) 107 { if (match_sig <= bsignatures[i]) percent = (match_sig / bsignatures[i]) * 100.0; else percent = (bsignatures[i] / match_sig) * 100.0; if (percent >= GlobalSearchFolder.threshold && percent >= small_dist) { small_dist = percent; index = i; } } } if (index == -1) return(null); return(image_files[index].getAbsolutePath()); } public double GetSignature(BufferedImage this_image) { double signature; int x, y, pixel_count; pixel_count = 0; signature = 0; 108 for (x = 0; x < this_image.getWidth(); x++) { for (y = 0; y < this_image.getHeight(); y++) { signature = signature + this_image.getRGB(x, y); pixel_count++; } } if (pixel_count == 0) return(0); if (signature < 0) signature = -signature; return(signature / pixel_count); } public void NewbuildImageList(String path) { // builds a list of images in the given directory. int i, n; File dir = new File(path); image_files = dir.listFiles(new JPEGImageFileFilter()); 109 rothers = new RenderedImage[image_files.length]; for (i = 0; i < image_files.length; i++) { try { n = getFaces(image_files[i]); if (n > 0) rothers[i] = Newrescale(ImageIO.read(image_files[i]), this_x, this_y, this_width, this_height); // rothers[i] = rescale(ImageIO.read(image_files[i])); else rothers[i] = rescale(ImageIO.read(image_files[i])); } catch (Exception e) { } } return; } public void buildImageList(String path) { 110 // builds a list of images in the given directory. int i; File dir = new File(path); image_files = dir.listFiles(new JPEGImageFileFilter()); rothers = new RenderedImage[image_files.length]; for (i = 0; i < image_files.length; i++) { try { rothers[i] = rescale(ImageIO.read(image_files[i])); } catch (Exception e) { } } return; } 111 public String NewfindImage(String source_image, int x, int y, int width, int height) { // finds the image with the closest match // return string with the image name. // build the image list File f_image = new File(source_image); double small_dist = 20000.0; int i, index = -1; String check_name; // find a close image try { // calculate the signature of the source image RenderedImage ref = Newrescale(ImageIO.read(f_image), x, y, width, height); signature = calcSignature(ref); // calculate the distance to the other images double[] distances = new double[image_files.length]; for (i = 0; i < image_files.length; i++) { 112 distances[i] = calcDistance(rothers[i]); } // now find the nearest image. for (i = 0; i < image_files.length; i++) { check_name = image_files[i].toString(); if (check_name.equalsIgnoreCase(GlobalSearchFolder.temp_name) != true) { if (distances[i] <= small_dist && distances[i] <= GlobalSearchFolder.threshold) { index = i; small_dist = distances[i]; } } } } catch (Exception e) { return(null); } // if a close image was found return it if (index != -1) return(image_files[index].toString()); 113 return(null); } public String findImage(String source_image) { // finds the image with the closest match // return string with the image name. // build the image list File f_image = new File(source_image); double small_dist = 20000.0; int i, index = -1; String check_name; // find a close image try { // calculate the signature of the source image RenderedImage ref = rescale(ImageIO.read(f_image)); signature = calcSignature(ref); 114 // calculate the distance to the other images double[] distances = new double[image_files.length]; for (i = 0; i < image_files.length; i++) { distances[i] = calcDistance(rothers[i]); } // now find the nearest image. for (i = 0; i < image_files.length; i++) { check_name = image_files[i].toString(); if (check_name.equalsIgnoreCase(GlobalSearchFolder.temp_name) != true) { if (distances[i] <= small_dist && distances[i] <= GlobalSearchFolder.threshold) { index = i; small_dist = distances[i]; } } } } catch (Exception e) { 115 return(null); } // if a close image was found return it if (index != -1) return(image_files[index].toString()); return(null); } private RenderedImage Newrescale(RenderedImage i, int x, int y, int width, int height) { RenderedImage crop_image, scaled_image; FileOutputStream os = null; // crop the image ParameterBlock crop_pb = new ParameterBlock(); crop_pb.addSource(i); crop_pb.add((float) this_x - (this_width / 2)); crop_pb.add((float) this_y - (this_width / 2)); crop_pb.add((float) this_width ); crop_pb.add((float) this_y - this_width); int xx, yy; 116 xx = i.getWidth(); yy = i.getHeight(); crop_image = null; try { crop_image = JAI.create("crop", crop_pb, null); os = new FileOutputStream("c:\\crap.png"); JAI.create("encode", crop_image, os, "PNG", null); } catch (Exception e) { int crap; crap = 0; } // scale the image. Image this_image = null; BufferedImage bi = null; Image new_scaled_image = null; 117 FileInputStream is; try { this_image = ImageIO.read(new File("c:\\crap.png")); new_scaled_image = this_image.getScaledInstance(300, 300, Image.SCALE_REPLICATE); bi = new BufferedImage(new_scaled_image.getWidth(null), new_scaled_image.getHeight(null), BufferedImage.TYPE_INT_RGB); Graphics2D g2 = bi.createGraphics(); g2.drawImage(new_scaled_image, 0, 0, null); ImageIO.write(bi, "PNG", new File("c:\\crap.png")); } catch (Exception e) { } float scaleW = ((float) baseSize) / crop_image.getWidth(); float scaleH = ((float) baseSize) / crop_image.getHeight(); 118 // Scales the original image ParameterBlock pb = new ParameterBlock(); pb.addSource(crop_image); pb.add(scaleW); pb.add(scaleH); pb.add(0.0F); pb.add(0.0F); pb.add(Interpolation.getInstance(Interpolation.INTERP_BICUBIC)); // Creates a new, scaled image and uses it on the DisplayJAI component scaled_image = JAI.create("scale", pb); return (bi); } private RenderedImage rescale(RenderedImage i) { // scale the image. float scaleW = ((float) baseSize) / i.getWidth(); float scaleH = ((float) baseSize) / i.getHeight(); // Scales the original image 119 ParameterBlock pb = new ParameterBlock(); pb.addSource(i); pb.add(scaleW); pb.add(scaleH); pb.add(0.0F); pb.add(0.0F); pb.add(new InterpolationNearest()); // Creates a new, scaled image and uses it on the DisplayJAI component return JAI.create("scale", pb); } private Color[][] calcSignature(RenderedImage i) { // calculate the signature for a given image // Get memory for the signature. Color[][] sig = new Color[5][5]; // For each of the 25 signature values average the pixels around it. // Note that the coordinate of the central pixel is in proportions. float[] prop = new float[] {1f / 10f, 3f / 10f, 5f / 10f, 7f / 10f, 9f / 10f}; for (int x = 0; x < 5; x++) for (int y = 0; y < 5; y++) sig[x][y] = averageAround(i, prop[x], prop[y]); 120 return sig; } private Color averageAround(RenderedImage i, double px, double py) { // Get an iterator for the image. RandomIter iterator = RandomIterFactory.create(i, null); // Get memory for a pixel and for the accumulator. double[] pixel = new double[3]; double[] accum = new double[3]; // The size of the sampling area. int sampleSize = 15; int numPixels = 0; // Sample the pixels. for (double x = px * baseSize - sampleSize; x < px * baseSize + sampleSize; x++) { for (double y = py * baseSize - sampleSize; y < py * baseSize + sampleSize; y++) { iterator.getPixel((int) x, (int) y, pixel); accum[0] += pixel[0]; accum[1] += pixel[1]; accum[2] += pixel[2]; numPixels++; 121 } } // Average the accumulated values. accum[0] /= numPixels; accum[1] /= numPixels; accum[2] /= numPixels; return new Color((int) accum[0], (int) accum[1], (int) accum[2]); } private double calcDistance(RenderedImage other) { // Calculate the distance for that image. Color[][] sigOther = calcSignature(other); // There are several ways to calculate distances between two vectors, // we will calculate the sum of the distances between the RGB values of // pixels in the same positions. double dist = 0; for (int x = 0; x < 5; x++) for (int y = 0; y < 5; y++) { int r1 = signature[x][y].getRed(); int g1 = signature[x][y].getGreen(); int b1 = signature[x][y].getBlue(); int r2 = sigOther[x][y].getRed(); 122 int g2 = sigOther[x][y].getGreen(); int b2 = sigOther[x][y].getBlue(); double tempDist = Math.sqrt((r1 - r2) * (r1 - r2) + (g1 - g2) * (g1 - g2) + (b1 - b2) * (b1 - b2)); dist += tempDist; } return dist; } } ShowImageInfo.java: import java.awt.image.ColorModel; import java.awt.image.DataBuffer; import java.awt.image.SampleModel; import java.io.File; import javax.media.jai.JAI; import javax.media.jai.PlanarImage; /** * This class display basic information about images. */ public class ShowImageInfo { 123 static StringBuilder sb = new StringBuilder(); static String printInfo(String s) { System.out.println("================================================= =="); // Display image data. First, the image size (non-JAI related). File image = new File(s); System.out.printf("File: %s Size in bytes: %d\n",s,image.length()); sb.append("File:"+s+"\nSize in bytes:"+image.length()+"\n"); // Open the image (using the name passed as a command line parameter) PlanarImage pi = JAI.create("fileload",s); // Now let's display the image dimensions and coordinates. System.out.print("Dimensions: "); System.out.print(pi.getWidth()+"x"+pi.getHeight()+" pixels"); sb.append("Dimensions: "+pi.getWidth()+"x"+pi.getHeight()+" pixels\n"); // Remember getMaxX and getMaxY return the coordinate of the next point! System.out.println(" (from "+pi.getMinX()+","+pi.getMinY()+ " to " +(pi.getMaxX()-1)+","+(pi.getMaxY()-1)+")"); sb.append(" (from "+pi.getMinX()+","+pi.getMinY()+ " to " +(pi.getMaxX()-1)+","+(pi.getMaxY()-1)+")\n"); System.out.print(pi.getTileWidth()+"x"+pi.getTileHeight()+" pixels"); 124 // Display info about the SampleModel of the image. SampleModel sm = pi.getSampleModel(); System.out.print("Data type: "); sb.append("Data type: "); switch(sm.getDataType()) { case DataBuffer.TYPE_BYTE: break; } { System.out.println("byte"); sb.append("byte\n"); case DataBuffer.TYPE_SHORT: { System.out.println("short");sb.append("short\n"); break; } case DataBuffer.TYPE_USHORT: { System.out.println("ushort");sb.append("ushort\n"); case DataBuffer.TYPE_INT: break; } break; } { System.out.println("int"); sb.append("int\n"); case DataBuffer.TYPE_FLOAT: { System.out.println("float");sb.append("float\n"); case DataBuffer.TYPE_DOUBLE: sb.append("double\n"); break; } break; } { System.out.println("double"); case DataBuffer.TYPE_UNDEFINED: { System.out.println("undefined");sb.append("undefined\n"); break; } } // Display info about the ColorModel of the image. ColorModel cm = pi.getColorModel(); if (cm != null) { System.out.println("Number of color components: "+cm.getNumComponents()); 125 sb.append("Number of color components: "+cm.getNumComponents()+"\n"); System.out.print("Colorspace components' names: "); sb.append("Colorspace components' names: "); for(int i=0;i<cm.getNumComponents();i++) { System.out.print(cm.getColorSpace().getName(i)+" "); sb.append(cm.getColorSpace().getName(i)+" "); } sb.append("\n"); System.out.println(); System.out.println("Bits per pixel: "+cm.getPixelSize()); sb.append("Bits per pixel: "+cm.getPixelSize()+"\n"); } else { System.out.println("No color model."); sb.append("No color model."); } return sb.toString(); } } FdvideoTest.java: import javax.media.*; 126 import java.io.File; public class fdVideoTest { // class to open and start processing a video static void receiveWebCam(String SearchFolder) { // this is for a webcam. CaptureDeviceInfo device; MediaLocator ml = null; FrameAccess fa = new FrameAccess(); //initiates the camera device = CaptureDeviceManager.getDevice("vfw:Microsoft WDM Image Capture (Win32):0"); //gets the location of the device, is needed for the player ml = device.getLocator(); // set the threshold for image matches for searching GlobalSearchFolder.threshold = 75.0; GlobalSearchFolder.webcam = true; 127 // open the the webcam fa.open(ml, SearchFolder); } static void receiveVideo(String FilePath, String SearchFolder) { MediaLocator ml = null; // this is for a file File f = new File(FilePath); FrameAccess fa = new FrameAccess(); try { ml = new MediaLocator(f.toURL()); } catch (Exception e) { } // open the the video GlobalSearchFolder.threshold = 100.0; GlobalSearchFolder.webcam = false; 128 fa.open(ml, SearchFolder); } } Fdlibjni.java: public class fdlibjni { static { System.loadLibrary("fdlibjni"); } native void detectfaces(byte[] imagedata, int imagewidth, int imageheight, int threshold); native int getndetections(); native int getdetectionx(int nr); native int getdetectiony(int nr); native int getdetectionwidth(int nr); } Fdtest.java: import java.net.URL; 129 import java.awt.Component; import java.awt.Graphics; import java.awt.image.*; import java.awt.Color; import java.io.File; import javax.imageio.*; import javax.swing.*; public class fdtest extends ImageIcon { static fdlibjni fdlib; static int n; static JFrame jf, jf1; static JPanel panel; static int w, h, threshold; static byte[] img; static StringBuilder sb = new StringBuilder(); public fdtest(URL url) { super(url); } static void receiveImage(String filepath, String searchfolder) { 130 //RECEIVE JPEG IMAGE FILE PATH jf = new JFrame(); jf1 = new JFrame(); panel = new JPanel(); fdlib = new fdlibjni(); threshold = 0; SearchImages search_images; String close_image; GlobalSearchFolder.threshold = 100.0; try { System.out.println("call"); //READ THE IMAGE BufferedImage bi = ImageIO.read(new File(filepath)); w = bi.getWidth(); h = bi.getHeight(); System.out.println("image is " + w + "x" + h + " pixels"); img = new byte[w*h]; for (int y=0, i=0; y<h; y++) { for (int x=0; x<w; x++, i++) { 131 //GET RGB VALUE TO FIND PIXEL COLOR int c = bi.getRGB(x,y); int r, g, b; //FIND RED COLOR r = (c&0x00ff0000)>>16; //FIND GREEN g = (c&0x0000ff00)>>8; //BLUE COLOR b = (c&0x000000ff); //CONVERT INTO GRAYSCALE COLOR img[i] = (byte) (.3*r + .59*g + .11*b); } } System.out.println("detecting with threshold " + threshold); fdlib.detectfaces(img, w, h, threshold); //GET THE COLOR VALUE AND DETECT IMAGE n = fdlib.getndetections(); if (n==1) System.out.println("1 face found."); else System.out.println(n + " faces found."); java.io.File f = new java.io.File(filepath); 132 //SET DETECTED IMAGE IN LABEL fdtest pi = new fdtest(f.toURL()); JLabel l = new JLabel(pi); String infoImg ="File Name: "+f.getName() +"\nFile Path: "+f.getPath() +"\nTotal Numer Of Faces Found: "+n; System.out.println(infoImg); HomeFile.imageInformation(infoImg); panel.add(l); jf.getContentPane().add(panel); jf.pack(); jf.setTitle("Face Detection Interface"); jf.setVisible(true); if (n > 0 && searchfolder != null) { // find a similiar image in the search folder search_images = new SearchImages(); search_images.buildImageList(searchfolder); close_image = search_images.findImage(filepath); if (close_image != null) { java.io.File fx = new java.io.File(close_image); 133 fdtest pix = new fdtest(fx.toURL()); JLabel lx = new JLabel(pix); panel.add(lx); jf1.getContentPane().add(panel); jf1.pack(); jf1.setTitle("Face Detection Interface"); jf1.setVisible(true); int index_1, index_2; index_1 = close_image.lastIndexOf('\\'); index_2 = close_image.lastIndexOf('.'); String this_name = close_image.substring(index_1 + 1, index_2); String search_info = "\n"; search_info = search_info + this_name + " was identified as a match."; HomeFile.imageInformation(search_info); } else { HomeFile.imageInformation("Nobody was identified as a match."); } } 134 } catch(Exception e) { System.out.println(e); } } public fdtest() { } public void paintIcon(Component c, Graphics g, int x, int y) { super.paintIcon(c,g,x,y); for (int i=0; i<n; i++) { //GET X Y AXIS System.out.println("x:" + fdlib.getdetectionx(i) + " y:" + fdlib.getdetectiony(i) + " size:" + fdlib.getdetectionwidth(i)); //SET RECT COLOR AS WIHTE g.setColor(Color.WHITE); //DRAW RECTANGLE TO IDENTIFY FACE g.drawRect(fdlib.getdetectionx(i)fdlib.getdetectionwidth(i),fdlib.getdetectiony(i)fdlib.getdetectionwidth(i),fdlib.getdetectionwidth(i)*2,fdlib.getdetectionwidth(i)*2); //SET STRING TO COUNT FACE 135 g.drawString(String.valueOf(i+1), fdlib.getdetectionx(i)(fdlib.getdetectionwidth(i)-10), fdlib.getdetectiony(i)-(fdlib.getdetectionwidth(i)-10)); Color s=g.getColor(); if(192<=s.getGreen() && s.getGreen()<=255 && 192<=s.getBlue() && s.getBlue()<=255 && 192<=s.getRed() && s.getRed()<=255) { System.out.println("Person :"+(i+1)+"White"); sb.append("Person "+(i+1)+" : White\n"); } else if(128<=s.getGreen() && s.getGreen()<=192 && 128<=s.getBlue() && s.getBlue()<=192 && 128<=s.getRed() && s.getRed()<=192) { System.out.println("Person :"+(i+1)+"Light White"); sb.append("Person "+(i+1)+" : Light White\n"); } else if(64<=s.getGreen() && s.getGreen()<=128 && 64<=s.getBlue() && s.getBlue()<=128 && 64<=s.getRed() && s.getRed()<=128) { System.out.println("Person :"+(i+1)+"Black"); sb.append("Person "+(i+1)+" : Black\n"); } } System.out.println("paintIcon"); } 136 }