SCHOOL OF ELECTRICAL ENGINEERING &
COMPUTER SCIENCE (SEECS)
CS470: Machine Learning
Project: Race Classifier
Group Members:
Muhammad Basit Raza (294216)
Muhammad Umair Alam (289917)
Submitted to:
Dr. Wajahat Hussain
Abstract:
In this project we have focused on the problem of developing an accurate and robust
model for the human skin detection and classification. Using DeepFace library a
neural network with back propagation training algorithm has been used to build the
classifier from a data set of skin pixels across different ethnic groups. It is difficult
to develop uniform method for the segmentation or detection of human skin detection
because the color tone of human skin is drastically varied for people from one region
to another. Skin detection plays an important role in a wide range of image
processing applications ranging from face detection, face tracking, gesture analysis,
and content-based image retrieval (CBIR) systems and to various human computer
interaction domains. In this project we gathered dataset from different search engines
to detect the proportion of different ethnic group in search result. Classifying the
dataset into four main classes i.e., white, Black, Brown and Light Brown. We used
Deepface library to analyze the images and predict the race of people in our dataset.
Literature review:
Skin texture and color are important signs that people use to understand variety of
culture-related aspects, such as health, ethnicity, age, attractiveness, wealth, and so
on. The presence of skin color in photos and videos indicates that people are present
in those media. As a result, substantial research in the area of expert and intelligent
systems has concentrated on skin identification in videos and photos during the
previous two decades. There are various tools used for human skin color and other
facial attributes detection these include OpenCV, SSD, Dlib, MTCNN, RetinaFace
and MediaPipe. The Neural Network (NN) classifies the image regions as a
collection of either skin or non-skin regions. Various approaches to skin modeling
are used in the literature. Here we give a brief review of the neural network models
for skin detection. Two types of skin models are used in the literature (Brown et al.,
2001) viz., symmetric and asymmetric. Symmetric model uses a single classifier for
both the classes whereas asymmetric model uses two separate classifiers for skin and
non-skin pixels that are separately trained using respective features. The neural
classifiers used in the literature either uses a symmetric model with single neuron in
the output layer or uses two separate neural networks (asymmetric model) for each
of the skin and non-skin classes. Most of the skin detection techniques discussed in
literature are used as a preprocessor for face detection and tracking systems.
However, when these techniques are used in real-time, it is crucial to follow time
deadlines and memory constraints. Sometimes, accuracy may need to be sacrificed
when the skin detection strategy is used only as a preprocessing step to face
detection, particularly in real time applications.
Dataset:
We collected the dataset using three different search engines which are google, Bing
and Duck Duck Go. From each search engine we collected 800 to 900 images for
each of the nationalities. For collection of data, we use a chrome-extension to
download images in ‘jpg’format.
Methodology:
We have used Deep Face library to analyze the images and predict the race of people
in our images. Deepface comes with a strong facial attribute analysis module
including age, gender, facial expression (including angry, fear, neutral, sad, disgust,
happy and surprise) and race (including Asian, white, middle eastern, Indian, latino
and black) predictions.
We first downloaded all images from different search engines including Google, Bing
and DuckDuckGo. Then we renamed all the images so we can run our code on all of
them using a for loop. We ran our DeepFace module (for race only) on each image
and counted the dominant race in different variables. Face detection and alignment
are important early stages of a modern face recognition pipeline. Experiments show
that just alignment increases the face recognition accuracy almost 1%. OpenCV,
SSD, Dlib, MTCNN, RetinaFace and MediaPipe detectors are wrapped in deepface
In our code we have used the detector named SSD, for fast results.
In the end we showed our results in the form of pie chart.
Filtering of the Noise in Dataset:
As the images are downloaded from search engines there is a lot of noise in our
dataset. These unwanted images reduces the accuracy of the results. Although we used
specific keywords to only get images for people but our search included other things
like flags, food and places. So in other to filter out the unwanted data we used OpenCV
library to detect the presence of human faces in the image. If the image does not
contain people the image is discarded and count as noise.
Following block of code is implemented to check for the presence of person in the
image.
#Import necessary packages
# Import libraries
import cv2
import numpy as np
import dlib
from google.colab import drive, files
drive.mount('/content/drive/')
path = "/content/drive/My Drive/ML-2/test5.jpg"
image = cv2.imread(path)
#print(image)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_front
alface_default.xml")
faces = faceCascade.detectMultiScale(
gray,
scaleFactor=1.3,
minNeighbors=3,
minSize=(30, 30)
)
print("Found {0} Faces!".format(len(faces)))
Code:
Following is our code to setup the DeepFace Classifier and test it on the dataset
form different search engines.
pip install deepface
import scipy.io as sio
from deepface import DeepFace
import cv2
import os
import matplotlib.pyplot as plt
GOOGLE_COLAB = True
path = ""
if GOOGLE_COLAB:
from google.colab import drive, files
drive.mount('/content/drive/')
path = "/content/drive/My Drive/ML-2/ML Project final/Pakista/Dataset"
count = 0
# Iterate directory
dir_path=path;
for pathe in os.listdir(dir_path):
# check if current path is a file
if os.path.isfile(os.path.join(dir_path, pathe)):
count += 1
print('File count:', count)
# Python 3 code to rename multiple
# files in a directory or folder
# importing os module
import os
# Function to rename multiple files
def main():
folder = path
for count, filename in enumerate(os.listdir(folder)):
dst = f"Img{str(count)}.jpg"
src =f"{folder}/{filename}" # foldername/filename, if .py file is outside
folder
dst =f"{folder}/{dst}"
# rename() function will
# rename all the files
os.rename(src, dst)
# Driver Code
if __name__ == '__main__':
# Calling main() function
main()
white=0;
light_brown=0;
brown=0;
black=0;
total_no=0;
backends = ['opencv', 'ssd', 'dlib', 'mtcnn', 'retinaface', 'mediapipe']
#Enter path of your test image
for i in range(count):
test_image=path+"Img"+str(i)+".jpg"
###### Checking number of faces in picture ########
#Import necessary packages
import dlib
image = cv2.imread(test_image)
#print(image)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
faceCascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalf
ace_default.xml")
faces = faceCascade.detectMultiScale(
gray,
scaleFactor=1.3,
minNeighbors=3,
minSize=(20, 20)
)
detfaces = int(format(len(faces)))
#print("Found {0} Faces!".format(len(faces)))
###################################################
if detfaces > 0:
total_no=total_no+1;
test_img = cv2.imread(test_image, cv2.IMREAD_UNCHANGED)[:,:,::-1]
#plt.imshow(test_img)
#plt.show
print(i)
obj= DeepFace.analyze(test_img,actions = ['race'],detector_backend = backends[1
],enforce_detection=False)
result = {key: obj[key] for key in obj.keys()
& {'dominant_race'}}
result= result.get('dominant_race')
#print("The Dominant race is",result)
if (result=='white'):
white=white+1;
elif (result=='asian'):
light_brown=light_brown+1;
elif (result=='middle eastern'):
brown=brown+1;
elif (result == 'black'):
black=black+1;
elif (result == 'latino hispanic'):
brown=brown+1;
elif (result=='indian'):
brown=brown+1;
save_path = '/content/drive/My Drive/ML-2/ML Project final/Pakistan/Output/'
name="Pakistani";
# defining labels
races = ['White','light Brown','Brown', 'Black']
# portion covered by each label
values = [white,light_brown,brown, black]
# color for each label
colors = ['r', 'y', 'g', 'b']
# plotting the pie chart
plt.pie(values, labels = races,
startangle=90, shadow = False,
radius = 2.2, autopct = '%1.2f%%')
# plotting legend
plt.legend(bbox_to_anchor=(2, 1), loc='upper left', borderaxespad=0)
plt.savefig(save_path+"pie_"+name+".png",bbox_inches='tight')
# showing the plot
plt.show()
fig = plt.figure()
ax = fig.add_axes([0,0,1,1])
langs = ['White','light Brown','Brown', 'Black']
value = [white,light_brown,brown, black]
ax.bar(langs,value)
plt.title("Google_"+name+".png")
plt.savefig(save_path+"bar_"+name+".png",bbox_inches='tight')
plt.show()
save_path = '/content/drive/My Drive/ML-2/ML Project final/Pakistan/Output/'
file_name = name+"_Google.txt"
completeName = os.path.join(save_path, file_name )
print(completeName)
file1 = open(completeName, "w")
file1.write("The total no of Images: "+str(total_no))
file1.write("\n")
file1.write("White People: "+str(white))
file1.write("\nBlack People: "+str(black))
file1.write("\nLight Brown People: "+str(light_brown))
file1.write("\nBrown People: "+str(brown))
file1.close()
Results:
Using the DeepFace module we classify the dataset in to four main classes i.e.,
white, Black, Brown and Light Brown. The dataset is collected from three search
engines Google, Bing and Duck Duck.Go. The results of our classifier are shown
using pie and bar graphs.
Following is the detailed analysis of our findings
Google Search:
Total images of 10 nationalities for Google = 8682
Correct images = 3781
Error images = 4901
Data-Noise = (4901 / 8681) x 100 = 56.45%
American
Chinese
French
German
Italian
Kenyan
Pakistani
Russian
South African
Spanish
Bing search:
Total images of 10 nationalities for Bing= 5272
Correct images = 2225
Error images = 3047
Data-Noise = (3047 / 5272) x 100 = 57.8%
Amrican
Chinese
German
Italian
Russian
Kenyan
Pakistani
French
Japanese
Spanish
Duck Duck Go search:
Total images of 10 nationalities for DuckDuckGo = 9187
Correct images = 4499
Error images = 4688
Data-Noise = (4688 / 9187) x 100 = 51.02%
American
Chinese
French
Germany
Italian
Kenyan
Pakistani
Russian
South African
Spanish
Conclusion:
A good skin classifier must be able to discriminate between skin and non-skin pixels
for a wide range of people with different skin types. Face detection and alignment are
very imported because they are the initial steps of any modern face recognition
pipeline. The dataset from search engines contains considerable among of noise. As
shown in the results, almost 50% of our dataset contained noise. So we can conclude
the dataset gathered through search engines is highly unreliable, requiring
preprocessing the datasets before beginning to train the Machine Learning model, as
we may end up spending hours on training only to discover that our model is highly
biased due to noisy data, resulting in lower accuracy on the testing dataset.
In this project we use DeepFace library for skin detection and prediction. DeepFace
library contains several detectors for face analysis. RetiaFace and MTCNN seem to
over perform in detection and alignment stages, but they are much slower. If speed of
your pipeline is more imported, then one should use OpenCv or SSD. On the other
hand, if accuracy is needed, then one should use RetinaFace or MTCNN.