OpenGL應用：實現人臉識別並貼紙的功能

人臉識別貼紙

整個處理過程大致分為3個步驟：
1、使用AVFoundation調用攝像頭采集視頻流獲得圖像信息
2、使用CoreImage庫判斷采集到的圖像信息中是否包含有人臉
3、將結果使用OpenGL渲染顯示到屏幕上

一、調用攝像頭采集視頻

self.captureSession = [[AVCaptureSession alloc] init];
[self.captureSession setSessionPreset:AVCaptureSessionPresetHigh];
        
AVCaptureDevice *captureDevice = nil;
NSArray *captureDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
for (AVCaptureDevice *device in captureDevices) {
    if (device.position == AVCaptureDevicePositionBack) {
       captureDevice = device;
       break;
     }
}
self.captureDeviceInput = [[AVCaptureDeviceInput alloc] initWithDevice:captureDevice error:nil];
        
if ([self.captureSession canAddInput:self.captureDeviceInput]) {
    [self.captureSession addInput:self.captureDeviceInput];
 }
        
self.captureDeviceOutput = [[AVCaptureVideoDataOutput alloc] init];
[self.captureDeviceOutput setAlwaysDiscardsLateVideoFrames:YES];
        
processQueue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_HIGH, 0);
[self.captureDeviceOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange] forKey:(id)kCVPixelBufferPixelFormatTypeKey]];
[self.captureDeviceOutput setSampleBufferDelegate:delegate queue:processQueue];
if ([self.captureSession canAddOutput:self.captureDeviceOutput]) {
    [self.captureSession addOutput:self.captureDeviceOutput];
}
        
AVCaptureConnection *captureConnection = [self.captureDeviceOutput connectionWithMediaType:AVMediaTypeVideo];
[captureConnection setVideoOrientation:AVCaptureVideoOrientationPortrait];
[self.captureSession startRunning];

獲得視頻幀：

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
    CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    [self.faceDetectionView displayPixelBuffer:pixelBuffer];
}

二、識別圖像中的人臉

CIImage *ciImage = [[CIImage alloc] initWithCVPixelBuffer:pixelBuffer];
NSString *accuracy = CIDetectorAccuracyLow;
NSDictionary *options = [NSDictionary dictionaryWithObject:accuracy forKey:CIDetectorAccuracy];
CIDetector *detector = [CIDetector detectorOfType:CIDetectorTypeFace context:nil options:options];
NSArray *featuresArray = [detector featuresInImage:ciImage options:nil];

得到的featuresArray便是識別的結果，是一個包含有CIFaceFeature對象的數組，我們可以使用獲得的結果判斷是否包含有人臉。

三、使用OpenGL渲染原始視頻幀和人臉位置貼圖

1.講我們要使用的貼圖圖片轉換成紋理數據，用於識別人臉后的紋理混合

CGImageRef spriteImage = [UIImage imageNamed:fileName].CGImage;
if (!spriteImage) {
    NSLog(@"Failed to load image %@", fileName);
    exit(1);
}

size_t width = CGImageGetWidth(spriteImage);
size_t height = CGImageGetHeight(spriteImage);

GLubyte *spriteData = (GLubyte *)calloc(width * height * 4, sizeof(GLubyte));

CGContextRef context = CGBitmapContextCreate(spriteData, width, height, 8, width * 4, CGImageGetColorSpace(spriteImage), kCGImageAlphaPremultipliedLast);
CGContextTranslateCTM(context, 0, height);
CGContextScaleCTM (context, 1.0, -1.0);
CGContextDrawImage(context, CGRectMake(0, 0, width, height), spriteImage);

CGContextRelease(context);

GLuint texture;
glActiveTexture(GL_TEXTURE2);
glGenTextures(1, &texture);
glBindTexture(GL_TEXTURE_2D, texture);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR );
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR );
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, (int32_t)width, (int32_t)height, 0, GL_RGBA, GL_UNSIGNED_BYTE, spriteData);
free(spriteData);
return texture;

2.渲染視頻幀，同時檢測有沒有人臉，如果有，計算出人臉位置，轉換坐標，將貼圖渲染上去。

- (void)displayPixelBuffer:(CVPixelBufferRef)pixelBuffer {
    if (pixelBuffer != NULL) {
        
        int width = (int)CVPixelBufferGetWidth(pixelBuffer);
        int height = (int)CVPixelBufferGetHeight(pixelBuffer);
        
        if (!_videoTextureCache) {
            NSLog(@"NO Video Texture Cache");
            return;
        }
        if ([EAGLContext currentContext] != _context) {
            [EAGLContext setCurrentContext:_context];
        }
        
        [self cleanUpTextures];
        
        glActiveTexture(GL_TEXTURE0);
        
        CVReturn err;
        err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                           _videoTextureCache,
                                                           pixelBuffer,
                                                           NULL,
                                                           GL_TEXTURE_2D,
                                                           GL_RED_EXT,
                                                           width,
                                                           height,
                                                           GL_RED_EXT,
                                                           GL_UNSIGNED_BYTE,
                                                           0,
                                                           &_lumaTexture);
        
        if (err) {
            NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
        }
        
        glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
        
        // UV-plane.
        glActiveTexture(GL_TEXTURE1);
        err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                           _videoTextureCache,
                                                           pixelBuffer,
                                                           NULL,
                                                           GL_TEXTURE_2D,
                                                           GL_RG_EXT,
                                                           width / 2,
                                                           height / 2,
                                                           GL_RG_EXT,
                                                           GL_UNSIGNED_BYTE,
                                                           1,
                                                           &_chromaTexture);
        if (err) {
            NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
        }
        
        glBindTexture(CVOpenGLESTextureGetTarget(_chromaTexture), CVOpenGLESTextureGetName(_chromaTexture));
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
        glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
        glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
        
        glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer);
        
        glViewport(0, 0, _backingWidth, _backingHeight);
        
    }
    
    glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
    glEnable(GL_BLEND);
    glClearColor(0, 0, 0, 1.0);
    glClear(GL_COLOR_BUFFER_BIT);
    
    glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer);
    [self.shaderManager useProgram];
    glUniform1i(glViewUniforms[UNIFORM_Y], 0);
    glUniform1i(glViewUniforms[UNIFORM_UV], 1);
    
    glUniformMatrix4fv(glViewUniforms[UNIFORM_ROTATE_MATRIX], 1, GL_FALSE, GLKMatrix4MakeXRotation(M_PI).m);
    
    GLfloat quadVertexData[] = {
        -1, -1,
        1, -1 ,
        -1, 1,
        1, 1,
    };
    
    // 更新頂點數據
    glVertexAttribPointer(glViewAttributes[ATTRIB_VERTEX], 2, GL_FLOAT, 0, 0, quadVertexData);
    glEnableVertexAttribArray(glViewAttributes[ATTRIB_VERTEX]);
    
    GLfloat quadTextureData[] =  { // 正常坐標
        0, 0,
        1, 0,
        0, 1,
        1, 1
    };
    
    glVertexAttribPointer(glViewAttributes[ATTRIB_TEXCOORD], 2, GL_FLOAT, GL_FALSE, 0, quadTextureData);
    glEnableVertexAttribArray(glViewAttributes[ATTRIB_TEXCOORD]);
    
    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
    
    [LYFaceDetector detectCVPixelBuffer:pixelBuffer completionHandler:^(CIFaceFeature *result, CIImage *ciImage) {
        if (result) {
            [self renderTempTexture:result ciImage:ciImage];
        }
    }];
    
    glBindRenderbuffer(GL_RENDERBUFFER, _renderBuffer);
    
    if ([EAGLContext currentContext] == _context) {
        [_context presentRenderbuffer:GL_RENDERBUFFER];
    }
}

3.轉換人臉坐標，計算需要渲染的貼圖坐標

- (void)renderTempTexture:(CIFaceFeature *)faceFeature ciImage:(CIImage *)ciImage {
    dispatch_semaphore_wait(_lock, DISPATCH_TIME_FOREVER);
    //得到圖片的尺寸
    CGSize ciImageSize = [ciImage extent].size;
    //初始化transform
    CGAffineTransform transform = CGAffineTransformScale(CGAffineTransformIdentity, 1, -1);
    transform = CGAffineTransformTranslate(transform,0,-ciImageSize.height);
    // 實現坐標轉換
    CGSize viewSize =self.layer.bounds.size;
    CGFloat scale = MIN(viewSize.width / ciImageSize.width,viewSize.height / ciImageSize.height);
    
    CGFloat offsetX = (viewSize.width - ciImageSize.width * scale) / 2;
    CGFloat offsetY = (viewSize.height - ciImageSize.height * scale) / 2;
    // 縮放
    CGAffineTransform scaleTransform = CGAffineTransformMakeScale(scale, scale);
    //獲取人臉的frame
    CGRect faceViewBounds = CGRectApplyAffineTransform(faceFeature.bounds, transform);
    // 修正
    faceViewBounds = CGRectApplyAffineTransform(faceViewBounds,scaleTransform);
    faceViewBounds.origin.x += offsetX;
    faceViewBounds.origin.y += offsetY;
    
    
    NSLog(@"face frame after:%@",NSStringFromCGRect(faceViewBounds));
    [self.textureManager useProgram];
    
    glBindTexture(GL_TEXTURE_2D, _myTexture);
    glUniform1i(glViewUniforms[UNIFORM_TEMP_INPUT_IMG_TEXTURE], 2);
    
    CGFloat midX = CGRectGetMidX(self.layer.bounds);
    CGFloat midY = CGRectGetMidY(self.layer.bounds);
    
    CGFloat originX = CGRectGetMinX(faceViewBounds);
    CGFloat originY = CGRectGetMinY(faceViewBounds);
    CGFloat maxX = CGRectGetMaxX(faceViewBounds);
    CGFloat maxY = CGRectGetMaxY(faceViewBounds);
    
    //貼圖頂點
    GLfloat minVertexX = (originX - midX) / midX;
    GLfloat minVertexY = (midY - maxY) / midY;
    GLfloat maxVertexX = (maxX - midX) / midX;
    GLfloat maxVertexY = (midY - originY) / midY;
    GLfloat quadData[] = {
        minVertexX, minVertexY,
        maxVertexX, minVertexY,
        minVertexX, maxVertexY,
        maxVertexX, maxVertexY,
    };
    
    glVertexAttribPointer(glViewAttributes[ATTRIB_TEMP_VERTEX], 2, GL_FLOAT, GL_FALSE, 0, quadData);
    glEnableVertexAttribArray(glViewAttributes[ATTRIB_TEMP_VERTEX]);
    
    GLfloat quadTextureData[] =  { // 正常坐標
        0, 0,
        1, 0,
        0, 1,
        1, 1
    };
    glVertexAttribPointer(glViewAttributes[ATTRIB_TEMP_TEXCOORD], 2, GL_FLOAT, GL_FALSE, 0, quadTextureData);
    glEnableVertexAttribArray(glViewAttributes[ATTRIB_TEMP_TEXCOORD]);
    
    glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
    dispatch_semaphore_signal(_lock);
}

這里只是簡單的利用了coreImage返回的CIFaceFeature對象提供的人臉坐標。其實CoreImage返回的CIFaceFeature對象可以提供很多信息，包括人臉坐標、左右眼是否睜開及對應位置、嘴的位置等，所以如果我們需要做更詳細的紋理貼圖可以分別轉換出眼睛、嘴巴的位置，然后使用我們想要的貼圖渲染到對應的紋理坐標系中即可。

這個demo有比較詳細的應用iOS CoreImage -- 人臉檢測/ 換背景/ 摳圖 /貼紙/ 實時視頻濾鏡

 

最后效果圖如下（這里為了簡單在得到的視頻幀中只處理了一個包含人臉的CIFaceFeature對象）