大道如青天，我獨不得出
前言
在上一篇iOS Core ML與Vision初識中，初步了解到了vision的作用，並在文章最后留了個疑問，就是類似下面的一些函數有什么用
- (instancetype)initWithCIImage:(CIImage *)image options:(NSDictionary<VNImageOption, id> *)options;
- (instancetype)initWithCVPixelBuffer:(CVPixelBufferRef)pixelBuffer options:(NSDictionary<VNImageOption, id> *)options;
在查閱一些資料后，最終通過這些函數得到了如下的效果
對，沒錯，這就是通過initWithCVPixelBuffer函數來實現的。當然vision的作用遠不於此，還有如下的效果
1、圖像匹配（上篇文章中的效果）
2、矩形檢測
3、二維碼、條碼檢測
4、目標跟蹤
5、文字檢測
6、人臉檢測
7、人臉面部特征檢測
由於對人臉識別比較感興趣，所以這里就主要簡單了解了下人臉部分，下面就針對人臉檢測和面部檢測寫寫
Vision支持的圖片類型
通過查看VNRequestHandler.h文件，我們可以看到里面的所有初始化函數，通過這些初始化函數，我們可以了解到支持的類型有：
1、CVPixelBufferRef
2、CGImageRef
3、CIImage
4、NSURL
5、NSData
Vision使用
在使用vision的時候，我們首先需要明確自己需要什么效果，然后根據想要的效果來選擇不同的類，最后實現自己的效果
1、需要一個RequestHandler，在創建RequestHandler的時候，需要一個合適的輸入源，及圖片類型
2、需要一個Request ，在創建Request 的時候，也需要根據實際情況來選擇，Request 大概有如下這么些
3、通過requestHandler將request聯系起來，然后得到結果
[handler performRequests:@[requset] error:&error];
4、處理結果VNObservation，在VNRequest的results數組中，包含了VNObservation結果，VNObservation也分很多種，這和你Request的類型是相關聯的
在完成上述步驟后，我們就可以根據結果來實現一些我們想要的效果
人臉矩形檢測
這里我們需要用到VNDetectFaceRectanglesRequest
requset = [[VNDetectFaceRectanglesRequest alloc] initWithCompletionHandler:completionHandler];
在得到結果后，我們需要處理下坐標
for (VNFaceObservation *faceObservation in observations) {
//boundingBox
CGRect transFrame = [self convertRect:faceObservation.boundingBox imageSize:image.size];
[rects addObject:[NSValue valueWithCGRect:transFrame]];
}
// 轉換Rect
- (CGRect)convertRect:(CGRect)boundingBox imageSize:(CGSize)imageSize{
CGFloat w = boundingBox.size.width * imageSize.width;
CGFloat h = boundingBox.size.height * imageSize.height;
CGFloat x = boundingBox.origin.x * imageSize.width;
CGFloat y = imageSize.height * (1 - boundingBox.origin.y - boundingBox.size.height);//- (boundingBox.origin.y * imageSize.height) - h;
return CGRectMake(x, y, w, h);
}
在返回結果中的boundingBox 中的坐標，我們並不能立即使用，而是需要進行轉換，因為這里是相對於image的一個比例，這里需要注意的是y坐標的轉換，因為坐標系的y軸和UIView的y軸是相反的。
最后就是通過返回的坐標進行矩形的繪制
+ (UIImage *)gl_drawImage:(UIImage *)image withRects:(NSArray *)rects
{
UIImage *newImage = nil;
UIGraphicsBeginImageContextWithOptions(image.size, NO, [UIScreen mainScreen].scale);
CGContextRef context = UIGraphicsGetCurrentContext();
CGContextSetLineCap(context,kCGLineCapRound); //邊緣樣式
CGContextSetLineJoin(context, kCGLineJoinRound);
CGContextSetLineWidth(context,2); //線寬
CGContextSetAllowsAntialiasing(context,YES); //打開抗鋸齒
CGContextSetStrokeColorWithColor(context, [UIColor redColor].CGColor);
CGContextSetFillColorWithColor(context, [UIColor clearColor].CGColor);
//繪制圖片
[image drawInRect:CGRectMake(0, 0,image.size.width, image.size.height)];
CGContextBeginPath(context);
for (int i = 0; i < rects.count; i ++) {
CGRect rect = [rects[i] CGRectValue];
CGPoint sPoints[4];//坐標點
sPoints[0] = CGPointMake(rect.origin.x, rect.origin.y);//坐標1
sPoints[1] = CGPointMake(rect.origin.x + rect.size.width, rect.origin.y);//坐標2
sPoints[2] = CGPointMake(rect.origin.x + rect.size.width, rect.origin.y + rect.size.height);//坐標3
sPoints[3] = CGPointMake(rect.origin.x , rect.origin.y + rect.size.height);
CGContextAddLines(context, sPoints, 4);//添加線
CGContextClosePath(context); //封閉
}
CGContextDrawPath(context, kCGPathFillStroke); //根據坐標繪制路徑
newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
return newImage;
}
效果如下
人臉特征識別
這里我們需要用到VNDetectFaceLandmarksRequest
requset = [[VNDetectFaceLandmarksRequest alloc] initWithCompletionHandler:completionHandler];
處理結果
for (VNFaceObservation *faceObservation in observations) {
//boundingBox
CGRect transFrame = [self convertRect:faceObservation.boundingBox imageSize:image.size];
[rects addObject:[NSValue valueWithCGRect:transFrame]];
}
pointModel.faceRectPoints = rects;
return pointModel;
}
- (GLDiscernPointModel *)handlerFaceLandMark:(NSArray *)observations image:(UIImage *)image
{
GLDiscernPointModel *pointModel = [[GLDiscernPointModel alloc] init];
NSMutableArray *rects = @[].mutableCopy;
for (VNFaceObservation *faceObservation in observations) {
VNFaceLandmarks2D *faceLandMarks2D = faceObservation.landmarks;
[self getKeysWithClass:[VNFaceLandmarks2D class] block:^(NSString *key) {
if ([key isEqualToString:@"allPoints"]) {
return ;
}
VNFaceLandmarkRegion2D *faceLandMarkRegion2D = [faceLandMarks2D valueForKey:key];
NSMutableArray *sPoints = [[NSMutableArray alloc] initWithCapacity:faceLandMarkRegion2D.pointCount];
for (int i = 0; i < faceLandMarkRegion2D.pointCount; i ++) {
CGPoint point = faceLandMarkRegion2D.normalizedPoints[i];
CGFloat rectWidth = image.size.width * faceObservation.boundingBox.size.width;
CGFloat rectHeight = image.size.height * faceObservation.boundingBox.size.height;
CGPoint p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * image.size.width, faceObservation.boundingBox.origin.y * image.size.height + point.y * rectHeight);
[sPoints addObject:[NSValue valueWithCGPoint:p]];
}
[rects addObject:sPoints];
}];
}
在這里，我們需要注意到landmarks 這個屬性，這是一個VNFaceLandmarks2D類型的對象，里面包含着許多面部特征的VNFaceLandmarkRegion2D對象，如：faceContour，leftEye，nose....分別表示面部輪廓、左眼、鼻子。這些對象中，又包含下面這么一個屬性
@property (readonly, assign, nullable) const CGPoint* normalizedPoints
這是一個包含該面部特征的的數組，所以我們可以通過下面的方式取出里面的坐標
CGPoint point = faceLandMarkRegion2D.normalizedPoints[i];
當然這里面也存在坐標的轉換，見上面代碼
最后也是畫線，代碼如下
+ (UIImage *)gl_drawImage:(UIImage *)image faceLandMarkPoints:(NSArray *)landMarkPoints
{
UIImage * newImage = image;
for (NSMutableArray *points in landMarkPoints) {
CGPoint sPoints [points.count];
for (int i = 0;i <points.count;i++) {
NSValue *pointValue = points[i];
CGPoint point = pointValue.CGPointValue;
sPoints[i] = point;
}
//畫線
UIGraphicsBeginImageContextWithOptions(newImage.size, NO, [UIScreen mainScreen].scale);
CGContextRef context = UIGraphicsGetCurrentContext();
CGContextSetLineCap(context,kCGLineCapRound); //邊緣樣式
CGContextSetLineJoin(context, kCGLineJoinRound);
CGContextSetLineWidth(context,2); //線寬
CGContextSetAllowsAntialiasing(context,YES); //打開抗鋸齒
// 設置翻轉
CGContextTranslateCTM(context, 0, newImage.size.height);
CGContextScaleCTM(context, 1.0, -1.0);
CGContextSetStrokeColorWithColor(context, [UIColor redColor].CGColor);
CGContextSetFillColorWithColor(context, [UIColor clearColor].CGColor);
CGContextDrawImage(context, CGRectMake(0, 0,newImage.size.width,newImage.size.height), newImage.CGImage);
CGContextBeginPath(context);
CGContextAddLines(context, sPoints,points.count);//添加線
CGContextClosePath(context); //封閉
CGContextDrawPath(context, kCGPathFillStroke); //根據坐標繪制路徑
newImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
}
return newImage;
}
效果如下
動態人臉矩形檢測
要動態來檢測，那么我們肯定需要通過相機來實時取出資源，然后再實現，所以我們這里選擇了AVCapture，關於相機的初始化及使用方法這里就不在累贅了，我們直接上代碼
在AVCaptureVideoDataOutputSampleBufferDelegate中，通過下面的方法
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
我們可以進行這么一個轉換
CVPixelBufferRef cvpixeBufferRef = CMSampleBufferGetImageBuffer(sampleBuffer);
然后通過
VNImageRequestHandler *handler = [[VNImageRequestHandler alloc] initWithCVPixelBuffer:cvpixeBufferRef options:@{}];
將相機返回的圖片與request進行關聯了。
后續操作如下
request = [[VNDetectFaceRectanglesRequest alloc] initWithCompletionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error) {
NSLog(@" 打印信息:%lu",request.results.count);
NSArray *vnobservations = request.results;
dispatch_async(dispatch_get_main_queue(), ^{
//先移除之前的矩形框
[self.rectLayers makeObjectsPerformSelector:@selector(removeFromSuperlayer)];
AVCaptureDevicePosition position = [[self.avInput device] position];
for (VNFaceObservation *faceObservation in vnobservations) {
//boundingBox
CGRect transFrame = [[GLTools sharedInstance] convertRect:faceObservation.boundingBox imageSize:self.view.frame.size];
//前置攝像頭的時候 記得轉換
if (position == AVCaptureDevicePositionFront){
transFrame.origin.x = self.view.frame.size.width - transFrame.origin.x - transFrame.size.width;
}
CALayer *rectLayer = [CALayer layer];
rectLayer.frame = transFrame;
rectLayer.borderColor = [UIColor purpleColor].CGColor;
rectLayer.borderWidth = 2;
[self.view.layer addSublayer:rectLayer];
[self.rectLayers addObject:rectLayer];
}
});
}];
在這里存在一個問題，就是攝像頭分為前后攝像頭，所以在前置攝像頭和后置攝像頭切換的時候，需要重新配置下
//需要重新進行配置輸出 特別是下面的輸出方向
AVCaptureConnection *captureConnection = [self.avOutput connectionWithMediaType:AVMediaTypeVideo];
if ([captureConnection isVideoOrientationSupported]) {
[captureConnection setVideoOrientation:AVCaptureVideoOrientationPortrait];
}
// 視頻穩定設置
if ([captureConnection isVideoStabilizationSupported]) {
captureConnection.preferredVideoStabilizationMode = AVCaptureVideoStabilizationModeAuto;
}
// 設置輸出圖片方向
captureConnection.videoOrientation = AVCaptureVideoOrientationPortrait;
還有個問題就是在坐標轉化的時候，前置攝像頭的x軸和UIView的x軸也是相反的，所以這里也需要在進行一次轉化
transFrame.origin.x = self.view.frame.size.width - transFrame.origin.x - transFrame.size.width;
效果如下
動態添加場景
關於動態添加場景，其實就像我們平時用的美顏相機那樣，在適當的位置添加些帽子、眼鏡等各種搞笑的圖片。這里我們還是需要用到AVCapture ，並且和動態添加矩形的方法類似，只是在request上和處理方式上不一樣
下面我們先看代碼
request = [[VNDetectFaceLandmarksRequest alloc] initWithCompletionHandler:^(VNRequest * _Nonnull request, NSError * _Nullable error) {
NSArray *vnobservations = request.results;
for (VNFaceObservation *faceObservation in vnobservations) {
VNFaceLandmarks2D *faceLandMarks2D = faceObservation.landmarks;
VNFaceLandmarkRegion2D *leftEyefaceLandMarkRegion2D = faceLandMarks2D.leftEye;
VNFaceLandmarkRegion2D *rightEyefaceLandMarkRegion2D = faceLandMarks2D.rightEye;
dispatch_async(dispatch_get_main_queue(), ^{
// //先移除之前的矩形框
// [self.rectLayers makeObjectsPerformSelector:@selector(removeFromSuperlayer)];
//
// AVCaptureDevicePosition position = [[self.avInput device] position];
//
// CGRect transFrame = [[GLTools sharedInstance] convertRect:faceObservation.boundingBox imageSize:self.view.frame.size];
// //前置攝像頭的時候 記得轉換
// if (position == AVCaptureDevicePositionFront){
// transFrame.origin.x = self.view.frame.size.width - transFrame.origin.x - transFrame.size.width;
// }
//
// CALayer *rectLayer = [CALayer layer];
// rectLayer.frame = transFrame;
// rectLayer.borderColor = [UIColor purpleColor].CGColor;
// rectLayer.borderWidth = 2;
// [self.view.layer addSublayer:rectLayer];
//
// [self.rectLayers addObject:rectLayer];
AVCaptureDevicePosition position = [[self.avInput device] position];
CGPoint sPoints[leftEyefaceLandMarkRegion2D.pointCount + rightEyefaceLandMarkRegion2D.pointCount];
NSMutableArray *pointXs = [[NSMutableArray alloc] init];
NSMutableArray *pointYs = [[NSMutableArray alloc] init];
for (int i = 0; i < leftEyefaceLandMarkRegion2D.pointCount; i ++) {
CGPoint point = leftEyefaceLandMarkRegion2D.normalizedPoints[i];
CGFloat rectWidth = self.view.bounds.size.width * faceObservation.boundingBox.size.width;
CGFloat rectHeight = self.view.bounds.size.height * faceObservation.boundingBox.size.height;
CGFloat boundingBoxY = self.view.bounds.size.height * (1 - faceObservation.boundingBox.origin.y - faceObservation.boundingBox.size.height);
CGPoint p = CGPointZero;
if (position == AVCaptureDevicePositionFront){
CGFloat boundingX = self.view.frame.size.width - faceObservation.boundingBox.origin.x * self.view.bounds.size.width - rectWidth;
p = CGPointMake(point.x * rectWidth + boundingX, boundingBoxY + (1-point.y) * rectHeight);
}else{
p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * self.view.bounds.size.width, boundingBoxY + (1-point.y) * rectHeight);
}
sPoints[i] = p;
[pointXs addObject:[NSNumber numberWithFloat:p.x]];
[pointYs addObject:[NSNumber numberWithFloat:p.y]];
}
for (int j = 0; j < rightEyefaceLandMarkRegion2D.pointCount; j ++) {
CGPoint point = rightEyefaceLandMarkRegion2D.normalizedPoints[j];
CGFloat rectWidth = self.view.bounds.size.width * faceObservation.boundingBox.size.width;
CGFloat rectHeight = self.view.bounds.size.height * faceObservation.boundingBox.size.height;
CGFloat boundingBoxY = self.view.bounds.size.height * (1 - faceObservation.boundingBox.origin.y - faceObservation.boundingBox.size.height);
CGPoint p = CGPointZero;
if (position == AVCaptureDevicePositionFront){
CGFloat boundingX = self.view.frame.size.width - faceObservation.boundingBox.origin.x * self.view.bounds.size.width - rectWidth;
p = CGPointMake(point.x * rectWidth + boundingX, boundingBoxY + (1-point.y) * rectHeight);
}else{
p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * self.view.bounds.size.width, boundingBoxY + (1-point.y) * rectHeight);
}
sPoints[leftEyefaceLandMarkRegion2D.pointCount + j] = p;
[pointXs addObject:[NSNumber numberWithFloat:p.x]];
[pointYs addObject:[NSNumber numberWithFloat:p.y]];
}
// for (UIView *view in self.view.subviews) {
// if ([view isKindOfClass:[UIImageView class]]) {
// [view removeFromSuperview];
// }
// }
//
// for (int i = 0; i < rightEyefaceLandMarkRegion2D.pointCount + leftEyefaceLandMarkRegion2D.pointCount; i++) {
// CGFloat x = sPoints[i].x;
// CGFloat y = sPoints[i].y;
// UIImageView *view = [[UIImageView alloc] initWithFrame:CGRectMake(x, y, 2, 2)];
// view.backgroundColor = [UIColor redColor];
// [self.view addSubview:view];
// }
//排序 得到最小的x和最大的x
NSArray *sortPointXs = [pointXs sortedArrayWithOptions:NSSortStable usingComparator:
^NSComparisonResult(id _Nonnull obj1, id _Nonnull obj2) {
int value1 = [obj1 floatValue];
int value2 = [obj2 floatValue];
if (value1 > value2) {
return NSOrderedDescending;
}else if (value1 == value2){
return NSOrderedSame;
}else{
return NSOrderedAscending;
}
}];
NSArray *sortPointYs = [pointYs sortedArrayWithOptions:NSSortStable usingComparator:
^NSComparisonResult(id _Nonnull obj1, id _Nonnull obj2) {
int value1 = [obj1 floatValue];
int value2 = [obj2 floatValue];
if (value1 > value2) {
return NSOrderedDescending;
}else if (value1 == value2){
return NSOrderedSame;
}else{
return NSOrderedAscending;
}
}];
UIImage *image =[UIImage imageNamed:@"eyes"];
CGFloat imageWidth = [sortPointXs.lastObject floatValue] - [sortPointXs.firstObject floatValue] + 40;
CGFloat imageHeight = (imageWidth * image.size.height)/image.size.width;
self.glassesImageView.frame = CGRectMake([sortPointXs.firstObject floatValue]-20, [sortPointYs.firstObject floatValue]-5, imageWidth, imageHeight);
});
}
}];
由於時間關系，代碼有點亂，將就將就
先說說思路，我是想動態添加一個眼鏡的，所以我必須先得到兩個眼睛的位置，然后在計算出兩個眼睛的寬高，最后適當的調整眼鏡的大小，再動態的添加上去
這里必須要說的一個問題，就是我在實現過程中遇到的---坐標
首先是y坐標，如果還是按照靜態圖片的那種獲取方式，那么得到的結果將會是完全相反的。
faceObservation.boundingBox.origin.y * image.size.height + point.y * rectHeight
這里我做了 一個假設，估計是由於攝像機成像的原因造成的，所以必須反其道而行，於是我如下改造了下
CGFloat boundingBoxY = self.view.bounds.size.height * (1 - faceObservation.boundingBox.origin.y - faceObservation.boundingBox.size.height);
p = CGPointMake(point.x * rectWidth + faceObservation.boundingBox.origin.x * self.view.bounds.size.width, boundingBoxY + (1-point.y) * rectHeight);
從中可以看到，所有的point.y都用1減去了，這個試驗的過程有點惱火，我還沒怎么相通，若有知道的，希望可以告訴我下，當然我也會再研究研究。
再說完y坐標后，就是x坐標了，x坐標在前置攝像頭的時候一切正常，然而在切換成后置攝像頭的時候，又反了。心累啊，所以沒辦法，我就只要加判斷，然后進行測試，有了如下代碼
CGFloat boundingX = self.view.frame.size.width - faceObservation.boundingBox.origin.x * self.view.bounds.size.width - rectWidth;
最后終於大功告成！
效果就是文章最頂的那個效果
項目文件截圖如下
注意
1、在使用過程中，我發現當檢測圖片的時候內存和cpu的消耗還是很高的，比如我的5s就成功的崩潰過.....
2、圖片方向是有要求的....
- (instancetype)initWithCVPixelBuffer:(CVPixelBufferRef)pixelBuffer options:(NSDictionary<VNImageOption, id> *)options;
/*!
@brief initWithCVPixelBuffer:options creates a VNImageRequestHandler to be used for performing requests against the image passed in as buffer.
@param pixelBuffer A CVPixelBuffer containing the image to be used for performing the requests. The content of the buffer cannot be modified for the lifetime of the VNImageRequestHandler.
@param orientation The orientation of the image/buffer based on the EXIF specification. For details see kCGImagePropertyOrientation. The value has to be an integer from 1 to 8. This superceeds every other orientation information.
@param options A dictionary with options specifying auxilary information for the buffer/image like VNImageOptionCameraIntrinsics
*/
- (instancetype)initWithCVPixelBuffer:(CVPixelBufferRef)pixelBuffer orientation:(CGImagePropertyOrientation)orientation options:(NSDictionary<VNImageOption, id> *)options;
通過對比上面兩個函數，我們可以發現，多了一個CGImagePropertyOrientation 類型的參數，沒錯，這就是指定傳入圖片的方向，如果指定了方向，而圖片方向卻不一致，那么恭喜你，檢測不出來....這里我用的都是第一個方法，及沒有參數，好像默認是up的。
iOS 11之Vision人臉檢測
注：本文著作權歸作者，由demo大師代發，拒絕轉載，轉載需要作者授權