Overview
Product recommendation is an important feature in beauty industry. When it comes to makeup recommendation, recommended products should be personalized depending on the users skin tone. To make the estimation process simple, no special devices should be used, but device used should be the one evey person has – a mobile phone. When selfie images are taken with a mobile phone, the image may appear good, but the white-balance is off and the colors are affected by the environment lighting. These problems in image acquisiton and the task of estimating the skin color together make this a complex task.
Goals
Develop a software solution that takes a selfie image as the input and estimates the skin tone of the person that took the selfie. Color estimation should be one of the classes in the MONK skin color scale. The color estimation should be the same for one person no matter the lighting present in the image.
Solution
Train a machine learning model that will correct the image by doing auto white-balancing and remove any environmental lighting from the scene. Training dataset consisted of images taken under colored lightning and with bad white-balancing and corrected images. Developed model was used as the preprocessing step for incoming photographs. Face segmentation model was used to acquire face region and do skin tone classification on that region. Since this step requires a dataset with specific annotations and images with human faces, which is a privacy issue, we created a pipeline for synthetic creation of 3D humans and images with skin annotations.
Results
- Solved the problem of misrepresentation of some skin tones (previously it worked for a couple of races only, now it works well for all).
- Outperformed other solutions when images are taken under bad lighting conditions.
