Facial landmark detection using Dlib (left) and CLM-framework (right).
Who sees the human face correctly: the photographer, the mirror, or the painter? — Pablo Picasso
If Picasso was alive today, he would have definitely added one more professional to that list — a computer vision engineer!
As computer vision engineers and researchers we have been trying to understand the human face since the very early days. The most obvious application of facial analysis is Face Recognition. But to be able to identify a person in an image we first need to find where in the image a face is located. Therefore, face detection — locating a face in an image and returning a bounding rectangle / square that contains the face — was a hot research area. In 2001, Paul Viola and Michael Jones pretty much nailed the problem with their seminal paper titled “Rapid Object Detection using a Boosted Cascade of Simple Features.” In the early days of OpenCV and to some extent even now, the killer application of OpenCV was a good implementation of the Viola and Jones face detector.
Once you have a bounding box around the face, the obvious research problem is to see if you can find the location of different facial features ( e.g. corners of the eyes, eyebrows, and the mouth, the tip of the nose etc ) accurately. Facial feature detection is also referred to as “facial landmark detection”, “facial keypoint detection” and “face alignment” in the literature, and you can use those keywords in Google for finding additional material on the topic.
Applications of Facial Keypoint Detection
There are several interesting applications of keypoint detection in human faces. A few of them are listed below.
Facial feature detection improves face recognition
Facial landmarks can be used to align facial images to a mean face shape, so that after alignment the location of facial landmarks in all images is approximately the same. Intuitively it makes sense that facial recognition algorithms trained with aligned images would perform much better, and this intuition has been confirmed by many research papers.
Head pose estimation
Once you know a few landmark points, you can also estimate the pose of the head. In other words you can figure out how the head is oriented in space, or where the person is looking. E.g. CLM-Framework described in this post also returns the head pose.
Face Morphing
Facial landmarks can be used to align faces that can then be morphed to produce in-between images. An example is shown in Figure 1.
Figure 1. A morph between President Bush and the The Governator!
Virtual Makeover
At my company ( TAAZ.com ) we had written our own facial landmark detector. The detected landmarks were used to the calculate contours of the mouth, eyes etc. to render makeup virtually. An example is shown in Figure 2.
Figure 2. Landmark detection for virtual makeover.
Face Replacement
If you have facial feature points estimated on two faces, you can align one face to the other, and then seamlessly clone one face onto the other. You can also do something goofy like this
https://auduno.github.io/clmtrackr/examples/facesubstitution.html
In a previous post, we showed how to use facial features to predict facial attractiveness.
Clearly, the ability to detect facial features in images and videos open up possibilities of a ton of interesting applications. Let us now get our hands dirty with some tools that will allow us to do this.
Facial Feature Detection & Tracking Libraries
There has been a flurry of activity in this area in the last 5 years. Part of the reason for this activity is the availability of large annotated datasets like LFPW and Helen. I have listed a bunch of papers in the next section. However, I do not recommend implementing these papers from scratch because now we have access to high quality open source implementations of some of these papers.
In the video below, you can see two of the libraries, Dlib and CLM-framework in action.
Dlib ( C++ / Python )
Dlib is a collection of miscellaneous algorithms in Machine Learning, Computer Vision, Image Processing, and Linear Algebra. Most of the library is just header files that you can include in your C++ application. Oh you prefer python ? No problem, it has a python API as well.
I personally like Dlib more than any other facial feature detection & tracking library because the code is very clean, well documented, the license permits use in commercial applications, the algorithm they have chosen to implement is very fast and accurate, and you can easily integrate the library in your C++ project by just including the header files.
How to compile Dlib ?
- Download a copy from github
1
git clone https://github.com/davisking/dlib.git - Build examples ( OSX / Linux )
12345
cd dlib/examplesmkdir buildcd buildcmake ..cmake --build . --config ReleaseThese examples are a great way to start using dlib. Make a copy of an example cpp file, modify it, modify examples/CMakeLists.txt and compile again using the instructions above. Easy!
- Compile dlib python module
12
cd dlib/python_examples./compile_dlib_python_module.bat - Set PYTHONPATH environment variable
12
# Put the following line in .bashrc or .profileexport PYTHONPATH=/path/to/dlib/python_examples:$PYTHONPATH - Test python module
1
python -c "import dlib"
If the above line does not give an error, you are all set.
In case you run into any compilation issues, there are additional instructions at Dlib.net
How to run Dlib’s facial landmark detector ?
After you have built the examples, to run the facial landmark detector using a webcam, do the following.
|
1
2
3
4
5
|
cd examples/build/
#Download the face landmark model
tar xvjf shape_predictor_68_face_landmarks.dat.bz2
./webcam_face_pose_ex
|
If you want to run it on a single image, you can try
|
1
|
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg
|
CLM-Framework (C++)
CLM-framework, also known as the Cambridge Face Tracker, is a C++ library for facial keypoint detection and head pose estimation. You can see how well it works in the included video. Compiling this library on OSX was bit of a challenge but it was not too bad. The library depends on OpenCV 3 and requires X11.
There are two important ways in which Dlib beats CLM-Framework. First, DLib is much faster than CLM-Framework. Second, Dlib’s license allows you to use it in commercial applications. If I had to pick, I would use Dlib. Interestingly, CLM-Framework depends on Dlib!
How to compile CLM-Framework ?
Compiling CLM-Framework was a bit involved for OSX. For windows and linux there are detailed instructions here. For compiling version 1.3.0 on OSX, I used the instructions for linux but made the following changes.
Most of the dependencies were installed using brew.
In file CMakeLists.txt ( red text was replaced with green text ).
find_package( OpenCV 3.0 REQUIRED )
INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
INCLUDE_DIRECTORIES(/opt/X11/include)
In file exe/SimpleCLM/SimpleCLM.cpp
How to run CLM-Framework’s Facial Feature Detector ?
After compiling CLM-Framework, the executables are in the bin directory. For the webcam demo shown in this post, you can use
|
1
|
bin/SimpleCLM
|
Face++ ( FacePlusPlus ) : Web API
One of the best implementations of facial landmark detection is by FacePlusPlus. They won the 300 Faces In-the-Wild Landmark Detection Challenge, 2013. They provide an easy to use API. The problem is that you need to upload an image to their servers and that raises a lot of privacy concerns. But if privacy is not an issue, Face++ is very good option. You can see a demo at
http://www.faceplusplus.com/demo-landmark/
Facial Feature Detection Research
Many different approaches have been used to solve this problem and it is difficult to summarize them in a blog post. I am simply linking to some important papers ( with major bias toward recent work ) for people who want to investigate more.
- Active Appearance Model (AAM) by T. Cootes, G. Edwards and C. J. Taylor. [1998]
- Face Alignment through Subspace Constrained Mean-Shifts by Jason M. Saragih, Simon Lucey and Jeffrey F. Cohn. [2009]
- Localizing Parts of Faces Using a Consensus of Exemplars by Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, Neeraj Kumar [ 2011 ]
- Face Alignment by Explicit Shape Regression by Xudong Cao Yichen Wei Fang Wen Jian Sun [2012]
- Supervised Descent Method and Its Applications to Face Alignment by Xuehan Xiong and Fernando De la Torre [2013]
- Constrained Local Neural Fields for robust facial landmark detection in the wild by Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. [2013]
- Extensive Facial Landmark Localization with Coarse-to-fine Convolutional Network Cascade by Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. [2013]
- Face alignment at 3000 fps via regressing local binary features by S Ren, X Cao, Y Wei, J Sun. [2014]
- Facial Landmark Detection by Deep Multi-task Learning by Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. [2014]
- One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan. [2014]
Subscribe
If you liked this article, please subscribe to our newsletter and receive a free
Computer Vision Resource guide. In our newsletter we share OpenCV tutorials and examples written in C++/Python, and Computer Vision and Machine Learning algorithms and news.
中文翻譯
源地址：http://blog.csdn.net/xiamentingtao/article/details/50908190
原文地址：http://www.learnopencv.com/facial-landmark-detection/#comment-2471797375
作為計算機視覺研究員，我們很早就開始研究人臉。人臉分析領域最廣為人知的就是人臉識別（face recognition）.但是為了識別一幅圖像中的人臉，我們首先必須要找到圖像中人臉的位置。因此人臉檢測（face detection)-定位一幅圖像中的人臉並且返回一個包圍人臉的矩形或者正方形（bounding rectangle/square）是一個熱門的研究領域。2001年，Paul Viola 和Michael Jones 發表了史詩級論文<< “Rapid Object Detection using a Boosted Cascade of Simple Features.>>.在OpenCV早期甚至某種程度下在現在，OpenCV的致命武器就是對
Viola and Jones face detector的一個比較好的實現。
一旦你找到了人臉附近的包圍盒，最顯然的研究當然是准確識別人臉不同特征的位置（比如，眼角、瞳孔、嘴巴、鼻子等）。人臉特征檢測（face feature detection）也稱為 “facial landmark detection”, “facial keypoint detection” and “face alignment”，你可以在Google找到類似的文獻。
Facial Keypoint Detection
人臉關鍵點檢測有很多應用。如下做了一些列舉：
Facial feature detection improves face recognition
人臉特征點可以被用來將人臉對齊到平均人臉（mean face shape）,這樣在對齊之后所有圖像中的人臉特征點的位置幾乎是相同的。直觀上來看，用對齊后的圖像訓練的人臉識別算法更加有效，這個直覺已經被很多論文驗證。
Head pose estimation
一旦你知道了一些特征點的位置，你也可以估計頭部的姿勢。換句話說，你可以解決頭部在空間中的定向問題，或者通俗的講就是人朝那里看的問題。
Face Morphing （人臉變形）
人臉特征點可以對齊人臉，這樣可以生成兩張人臉的中間圖像。如下圖： 
Virtual Makeover（虛擬化妝）
在我的公司
我們已經寫了自己的人臉特征點檢測器。檢測出的特征點被用來計算嘴的輪廓，眼睛等用來渲染虛擬化妝。Figure2z展示了這一效果: 
Face Replacement
如何兩張人臉的特征點已經估計出來了，你可以將一張人臉對齊到另一張人臉，並且可以無縫換臉。你也可以做像下面一樣傻瓜的事。
https://auduno.github.io/clmtrackr/examples/facesubstitution.html
先前的報告中，我們展示了如何使用人臉特征點去預測人臉的吸引力。
很明顯，在圖片和視頻上進行人臉特征點檢測為許多有趣的應用提供了很多的可能性。下面我們就將介紹一些有用的特征點檢測工具。
Facial Feature Detection & Tracking Libraries
過去五年來，這個領域很火，部分原因是大量可以用來訓練的數據如LFPW、Helen被提供。我在下一節列了很多論文。但是我不建議胡亂實現這些論文，因為已經有開源的實現。
下面的視頻中，你可以看到兩個庫Dlib和CLM-framework.
http://7xrqgw.com1.z0.glb.clouddn.com/dlib_clm.mp4
Dlib(C++/Python)
Dlib是機器學習，計算機視覺，圖像處理，線性代數中眾多算法的集合。庫中大多數是頭文件，你可以直接直接包含在C++應用中。或者你更喜歡Python?沒問題，他也有一個Python接口.
我個人更喜歡Dlib因為代碼是簡潔的，有大量的注釋，也可以被用來商用。他們選擇實現的算法是非常快的，並且是准確的，你可以很容易集成這個庫到你的C++工程中，而你需要做的僅僅是包含頭文件.
如何編譯Dlib?
- 從Github上下載：
git clone https://github.com/davisking/dlib.git- 1
- 建立Examples（OSX\Linux）
cd dlib/examples mkdir build cd build cmake .. cmake --build . --config Release- 1
- 2
- 3
- 4
- 5
這些例子是一個開始使用Dlib的非常好的方法。拷貝一個例子的cpp文件，修改它，修改examples/CMakeLists.txt 並且像上面一樣再一次編譯它。很容易吧！
3. 編譯dlib python 模塊
cd dlib/python_examples ./compile_dlib_python_module.bat- 1
- 2
- 設置 PYTHONPATH 環境變量
# Put the following line in .bashrc or .profile export PYTHONPATH=/path/to/dlib/python_examples:$PYTHONPATH - 1
- 2
- 3
- 測試python模塊
python -c "import dlib"- 1
如果以上都沒有問題的話，你就設置好了。
How to run Dlib’s facial landmark detector ?
當你編譯好examples后，為了在網絡攝像頭上運行人臉特征點檢測器，可以這樣做:
cd examples/build/
#Download the face landmark model wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 tar xvjf shape_predictor_68_face_landmarks.dat.bz2 ./webcam_face_pose_ex- 1
- 2
- 3
- 4
- 5
如果你想要在單個圖像上運行，你可以這樣試試:
./face_landmark_detection_ex shape_predictor_68_face_landmarks.dat faces/*.jpg - 1
- 2
CLM-Framework (C++)
CLM-framework,也被稱為劍橋人臉跟蹤器，是一個用來進行人臉特征點檢測和頭部姿勢估計的C++庫。你可以看看他在包含的video文件里工作的多么好啊！在OSX上編譯這個庫有點兒挑戰但是也不太難。庫依賴於OpenCV3和X11.
有兩個重要的事說明Dlib可以挑戰CLM-Framework。首先，Dlib比CLM-Framework更快。其次，Dlib的license允許你商用。如果要挑一個的，我會使用Dlib.有趣的是，CLM-Framework依賴於Dlib.
如何編譯CLM-Framework？
編譯CLM-Framework在OSX上有點兒復雜。對於Windows和linux，這里有一份詳細的說明.為了在OSX上編譯version 1.3.0,我使用了linux的指示，但是發生了很多改變。
許多依賴項可以使用brew安裝.
在文件CMakeLists.txt(如下划掉的被后面的取代)
find_package( OpenCV 3.0 REQUIRED )
find_package( OpenCV 3.0 REQUIRED HINTS /path/to/opencv )
INCLUDE_DIRECTORIES(${OpenCV_INCLUDE_DIRS})
INCLUDE_DIRECTORIES(/opt/X11/include)
在文件exe/SimpleCLM/SimpleCLM.cpp中
writerFace = VideoWriter(tracked_videos_output[f_n], CV_FOURCC(‘D’,’I’,’V’,’X’), 30, captured_image.size(), true);
writerFace = VideoWriter(tracked_videos_output[f_n], CV_FOURCC(‘M’,’P’,’4′,’V’), 15, captured_image.size(), true);
如何運行CLM-Framework人臉檢測器？
編譯后，可執行文件在bin路徑中.對於視頻中展現的網絡攝像頭Demo，你可以這樣使用：
bin/SimpleCLM- 1
Face++ ( FacePlusPlus ) : Web API
人臉特征點檢測最好的實現之一就是Face++.他們在300 Faces in-the-Wild
Landmark Detection Challenge,2013取得了冠軍。他們提供了一個易用的API。問題是你需要上傳一張圖片到他們的服務器，這個將帶來很多隱私上的擔憂。但是如果隱私不是問題的話，Face++是一個好的選擇。你可以在
http://www.faceplusplus.com/demo-landmark/
看到一個Demo.
Facial Feature Detection Research
許多不同的方法都可以用來解決這個問題。很難再博客中對其歸類。我簡單地列出了一些重要論文。
1. Active Appearance Model (AAM) by T. Cootes, G. Edwards and C. J. Taylor. [1998]
2. Face Alignment through Subspace Constrained Mean-Shifts by Jason M. Saragih, Simon Lucey and Jeffrey F. Cohn. [2009]
3. Localizing Parts of Faces Using a Consensus of Exemplars by Peter N. Belhumeur, David W. Jacobs, David J. Kriegman, Neeraj Kumar [ 2011 ]
4. Face Alignment by Explicit Shape Regression by Xudong Cao Yichen Wei Fang Wen Jian Sun [2012]
5. Supervised Descent Method and Its Applications to Face Alignment by Xuehan Xiong and Fernando De la Torre [2013]
6. Constrained Local Neural Fields for robust facial landmark detection in the wild by Tadas Baltrusaitis, Peter Robinson, and Louis-Philippe Morency. [2013]
7. Extensive Facial Landmark Localization with Coarse-to-fine Convolutional Network Cascade by Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang and Qi Yin. [2013]
8. Face alignment at 3000 fps via regressing local binary features by S Ren, X Cao, Y Wei, J Sun. [2014]
9. Facial Landmark Detection by Deep Multi-task Learning by Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. [2014]
10.One Millisecond Face Alignment with an Ensemble of Regression Trees by Vahid Kazemi and Josephine Sullivan. [2014]