Project Tango應該說是Google一試水AR的設備,其中Project Tango主要二個功能,一個是獲取深度信息,如MS的Kinect,有相當多的設備都有這個功能,二是第一人稱相對定位,這個就沒那么常見了,如果對這個設備有更深的興趣,可以看知乎上的這二個鏈接。
Google Project Tango獲取深度信息的原理是什么?
Project tango是如何僅憑自身攝像頭實現位置追蹤的?
在這就不仔細來說這個東東了,上面二個鏈接比我自己再來說篇好多了,Project Tango本身有Unity3D的包(googlesamples/tango-examples-unity)如果在Unity3D下開發,相應的東東都已經提供,還有一些實例,能夠很容易就開發基於Project Tango的功能出來。
UE4下就比較麻煩了,google沒有針對UE4做相應的包,不過,google提供針對安卓開發的項目,一種是Android Studio項目,一個是供JNI調用的C語言項目。
googlesamples/tango-examples-java
googlesamples/tango-examples-c
UE4針對移動平台感覺還是不那么友好,如Unity調用Android項目,大家順便一搜,都能搞定,而在UE4下引用安卓項目,如下是一個添加針對Android支持Google Play功能的committed。
UE4 Google Play support on Android
可以看到,有些復雜,特別針對我這種UE4與Android都不熟的人,只有想別辦法,首先我的需求並不復雜,只是在一個模型與現實重疊的空間利用Project Tango的Motion tracking功能行走,旋轉等,簡單來說,我現在的辦公室環境,利用3D建模做一個和辦公室一樣的模型,長寬都要對上,這樣利用Project Tango 的Tracking,我能只看屏幕也知道我在辦公室的那個位置,前面是否有障礙物,就如HTV vive的那二個像個小音箱的東東來檢測可活動區域一樣。
UE4本身的腳本就是C++語言,自然我就想到利用上面的tango c來做開發,如下主要記錄本文實踐這種方法遇到的一些問題。
首先安裝UE4的安卓開發相關所有軟件,UE4已經幫你差不多都搞好。安裝安卓軟件開發工具包(SDK) ,然后對照這個鏈接下的UE4安卓快速入門自己測試,一個簡單的UE4 安卓程序發出來就沒問題了。
tango-examples-c里很簡單,一個so文件,相當於win平台中的動態鏈接庫文件,二是一個頭文件tango_client_api.h.現在就很簡單了,相當於平常我們寫C++程序一樣,引入動態鏈接庫,然后添加頭文件就可,但是UE4中,編譯都是靠對應目錄下的cs文件編譯,我們需要讓UE4的規則來引入庫與頭文件。
按照UE4大家默認的一些目錄位置與命名,先把相應頭文件與庫放入一個ThirdParth文件夾,然后放入UE4工程文件件,與Source,Binaries等目錄平行,如下:

然后打開工程名.Build.cs文件,告訴編譯器我們需要引入的庫與文件。
public class Office_05 : ModuleRules { public Office_05(TargetInfo Target) { PublicDependencyModuleNames.AddRange(new string[] { "Core", "CoreUObject", "Engine", "InputCore", "Launch", "UMG" }); //PrivateDependencyModuleNames.AddRange(new string[] { "" }); //PublicIncludePaths.Add("Runtime/Launch/Public"); //PrivateIncludePaths.Add("Runtime/Launch/Private/Android"); if (Target.Platform == UnrealTargetPlatform.Android) { PublicIncludePaths.AddRange(new string[] { "Core" }); PublicIncludePaths.Add("C:/NVPACK/android-ndk-r10e/platforms/android-19/arch-arm/usr/include"); LoadBobsMagic(Target); } } private string ModulePath { get { return Path.GetDirectoryName(RulesCompiler.GetModuleFilename(this.GetType().Name)); } } private string ThirdPartyPath { get { return Path.GetFullPath(Path.Combine(ModulePath, "../../ThirdParty/")); } } public bool LoadBobsMagic(TargetInfo Target) { bool isLibrarySupported = false; if (Target.Platform == UnrealTargetPlatform.Android) { isLibrarySupported = true; string LibrariesPath = Path.Combine(ThirdPartyPath, "tango_client_api", "lib"); PublicAdditionalLibraries.Add(Path.Combine(LibrariesPath, "libtango_client_api.so")); } if (isLibrarySupported) { // Include path PublicIncludePaths.Add(Path.Combine(ThirdPartyPath, "tango_client_api", "include")); } //Definitions.Add(string.Format("WITH_LIBZPLAY_BINDING={0}", isLibrarySupported ? 1 : 0)); return isLibrarySupported; } }
PublicDependencyModuleNames我們新添加Launch與UMG,因為我們需要引用這二個庫,不然后面引用#include "Android/AndroidJNI.h"里的功能會告訴你沒有實現,還有一點特別注意,安卓的功能一定要包含預處理定義PLATFORM_ANDROID當中,當初特別二的以為編譯選項選擇Android后,就可以直接寫NDK代碼了,當然現在的編譯是由UE4來控制的,所以不用看VS中的錯誤列表,只需要看VS中的輸出,如果提示生成成功,沒有error,就可以在設備上發布了。
然后我們包裝一下tango_client_api.h里的功能,演示如何在安卓環境下調用JNIEnv,當前active等。
#pragma once #include "Components/TextRenderComponent.h" #include "Office_05.h" #include "MyCharacter.h" #include "TangoApp.h" #include "Engine/TextRenderActor.h" #include "Components/TextRenderComponent.h" #if PLATFORM_ANDROID #include "tango_client_api.h" #include "Android/AndroidApplication.h" #include "Android/AndroidJava.h" #include "Android/AndroidJNI.h" #endif /** * */ class OFFICE_05_API TangoApp { private: static class UTextRenderComponent* textRender; static FVector translation; static FQuat quat; public: TangoApp(); ~TangoApp(); static FVector& GetTranslation() { return translation; } static FQuat& GetQuat() { return quat; } static void SetTextRender(class UTextRenderComponent* tRender) { textRender = tRender; appendText(textRender, TEXT("VV")); } static void appendText(UTextRenderComponent* textCompent, const FString& value, bool overrid = false) { if (textCompent == nullptr) return; if (!overrid) { FText text = textCompent->Text; FString newText = text.BuildSourceString(); newText.Append(" "); newText.Append(value); textCompent->SetText(FText::FromString(newText)); } else { textCompent->SetText(FText::FromString(value)); } } #if PLATFORM_ANDROID static void InitTango() { JNIEnv* Env = FAndroidApplication::GetJavaEnv(); jint VersionJint = Env->GetVersion(); int8 Version = (int8)VersionJint; TangoApp::appendText(textRender, FString::FromInt(Version)); jobject currentActive = FAndroidApplication::GetGameActivityThis(); TangoErrorType type = TangoService_initialize(Env, currentActive); TangoApp::appendText(textRender, FString::FromInt((int)type)); //type = TangoService_setBinder(Env, currentActive); TangoApp::appendText(textRender, TEXT("A")); TangoApp::appendText(textRender, FString::FromInt((int)type)); auto tangoConfig = TangoService_getConfig(TANGO_CONFIG_DEFAULT); type = TangoConfig_setBool(tangoConfig, "config_enable_motion_tracking", true); TangoApp::appendText(textRender, FString::FromInt((int)type)); if (type == TANGO_SUCCESS) { TangoApp::appendText(textRender, TEXT("B")); } type = TangoConfig_setBool(tangoConfig, "config_enable_auto_recovery", true); TangoApp::appendText(textRender, FString::FromInt((int)type)); if (type == TANGO_SUCCESS) { TangoApp::appendText(textRender, TEXT("C")); } type = TangoConfig_setBool(tangoConfig, "config_enable_learning_mode", true); TangoApp::appendText(textRender, FString::FromInt((int)type)); if (type == TANGO_SUCCESS) { TangoApp::appendText(textRender, TEXT("D")); } //TangoApp::Launch(); //TangoApp::appendText(text->GetTextRender(), TangoApplication::getPackageName()); //uuid得不到,相應權限申請不成功,請看TangoApplication::Launch char* uuidList = NULL; type = TangoService_getAreaDescriptionUUIDList(&uuidList); TangoApp::appendText(textRender, FString::FromInt((int)type)); if (type == TANGO_SUCCESS) { int lenght = 0; for (int i = 0; i < 1000; i++) { if (uuidList[i] != 0) ++lenght; else { break; } } FString suuidList(uuidList); TangoApp::appendText(textRender, FString::FromInt(lenght)); } TangoCoordinateFramePair pair; pair.base = TANGO_COORDINATE_FRAME_START_OF_SERVICE; pair.target = TANGO_COORDINATE_FRAME_DEVICE; //用來驗證相應數據 //TangoPoseData* poseData = new TangoPoseData(); //pair.base = TANGO_COORDINATE_FRAME_IMU; //pair.target = TANGO_COORDINATE_FRAME_CAMERA_COLOR; //TangoService_getPoseAtTime(0, pair, poseData); //type = TangoService_connectOnTangoEvent(&TangoApp::onTangoConnectEvent); //TangoApp::appendText(textRender), TEXT("E")); //TangoApp::appendText(textRender, FString::FromInt((int)type)); type = TangoService_connectOnPoseAvailable(1, &pair, &TangoApp::TangoService_onPoseAvailable); TangoApp::appendText(textRender, TEXT("F")); TangoApp::appendText(textRender, FString::FromInt((int)type)); //FAppEventManager::GetInstance()->PauseRendering(); type = TangoService_connect(nullptr, tangoConfig); TangoApp::appendText(textRender, FString::FromInt((int)type)); //FAppEventManager::GetInstance()->ResumeRendering(); } static FString getPackageName() { if (JNIEnv* Env = FAndroidApplication::GetJavaEnv(true)) { auto getPackageMethod = FJavaWrapper::FindMethod(Env, FJavaWrapper::GameActivityClassID, "getPackageName", "()Ljava/lang/String;", false); jstring jsString = (jstring)FJavaWrapper::CallObjectMethod(Env, FJavaWrapper::GameActivityThis, getPackageMethod); check(jsString); const char * nativeName = Env->GetStringUTFChars(jsString, 0); FString ResultName = FString(nativeName); Env->ReleaseStringUTFChars(jsString, nativeName); Env->DeleteLocalRef(jsString); return ResultName; } return FString(); } static void Launch() { if (JNIEnv* Env = FAndroidApplication::GetJavaEnv(true)) { //申請ADF權限 auto intentClass = Env->FindClass("android/content/Intent"); auto Constructor = Env->GetMethodID(intentClass, "<init>", "()V"); auto intentObject = Env->NewObject(intentClass, Constructor); auto intentMethod = FJavaWrapper::FindMethod(Env, intentClass, "setClassName", "(Ljava/lang/String;Ljava/lang/String;)V", false); //auto putExtraMethod = FJavaWrapper::FindMethod(Env, intentClass, "putExtra", "(Ljava/lang/String;Ljava/lang/String;)V", false); /* FJavaWrapper::CallVoidMethod(Env, intentObject, intentMethod, "com.projecttango.tango", "com.google.atap.tango.RequestPermissionActivity"); FJavaWrapper::CallVoidMethod(Env, intentObject, putExtraMethod, "PERMISSIONTYPE", "ADF_LOAD_SAVE_PERMISSION"); auto startAMehtod = FJavaWrapper::FindMethod(Env, FJavaWrapper::GameActivityClassID, "startActivity", "(android/content/Intent;)V", false); */ //FJavaWrapper::CallVoidMethod(Env, currentActive, startAMehtod, intentObject); //check(object); //auto intentObject = Env->NewGlobalRef(object); Env->DeleteLocalRef(intentObject); } //auto intentMethod = FJavaWrapper::FindMethod(Env, intentClass, "setClassName", "(Ljava/lang/String;Ljava/lang/String;)V", false); //auto putExtraMethod = FJavaWrapper::FindMethod(Env, intentClass, "putExtra", "(Ljava/lang/String;Ljava/lang/String;)V", false); /* FJavaWrapper::CallVoidMethod(Env, intentObject, intentMethod, "com.projecttango.tango", "com.google.atap.tango.RequestPermissionActivity"); FJavaWrapper::CallVoidMethod(Env, intentObject, putExtraMethod, "PERMISSIONTYPE", "ADF_LOAD_SAVE_PERMISSION"); auto startAMehtod = FJavaWrapper::FindMethod(Env, FJavaWrapper::GameActivityClassID, "startActivity", "(android/content/Intent;)V", false); */ //FJavaWrapper::CallVoidMethod(Env, currentActive, startAMehtod, intentObject); //FJavaClassObject intentObject(FName("android/content/Intent"), "()V"); //auto intentMethod = intentObject.GetClassMethod("setClassName", "(Ljava/lang/String;Ljava/lang/String;)V"); //auto putExtraMethod = intentObject.GetClassMethod("putExtra", "(Ljava/lang/String;Ljava/lang/String;)V"); //intentObject.CallMethod(intentMethod, "com.projecttango.tango", "com.google.atap.tango.RequestPermissionActivity"); //intentObject.CallMethod(intentMethod, "PERMISSIONTYPE", "ADF_LOAD_SAVE_PERMISSION"); //auto startAMehtod = FJavaWrapper::FindMethod(Env, FJavaWrapper::GameActivityClassID, "startActivity", "(android/content/Intent;)V", false); //FJavaWrapper::CallVoidMethod(Env, currentActive, startAMehtod, intentObject.GetJObject()); //auto tangoClass = Env->GetObjectClass(currentActive); //auto tangoMethod = FJavaWrapper::FindMethod(FAndroidApplication::GetJavaEnv(true), FJavaWrapper::GameActivityClassID, "launchIntent", "(Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;I)V", false); //auto tangoMethod = FJavaWrapper::FindMethod(Env, tangoClass, "launchIntent", "(Ljava/lang/String;Ljava/lang/String;[Ljava/lang/String;I)V", false); //if (tangoMethod != nullptr) //{ // TangoApplication::appendText(text->GetTextRender(), TEXT("C")); // //使用方法請看AndroidJNI.cpp ->AndroidThunkCpp_Iap_QueryInAppPurchases // jobjectArray tArgs = (jobjectArray)Env->NewObjectArray(1, FJavaWrapper::JavaStringClass, NULL); // jstring StringValue = Env->NewStringUTF(TCHAR_TO_UTF8("PERMISSIONTYPE:ADF_LOAD_SAVE_PERMISSION")); // Env->SetObjectArrayElement(tArgs, 0, StringValue); // Env->DeleteLocalRef(StringValue); // FJavaWrapper::CallVoidMethod(Env, currentActive, tangoMethod, "com.projecttango.tango", // "com.google.atap.tango.RequestPermissionActivity", tArgs, 43); //} } static void onTangoConnectEvent(void* context, const TangoEvent* event) { appendText(textRender, TEXT("Connect"), true); appendText(textRender, FString::SanitizeFloat(event->timestamp)); } //Unity x左右,y上下,z前后 左手 //https://developers.google.com/project-tango/overview/coordinate-systems#project_tango_coordinate_frames //Tango START_OF_SERVICE x左右,y前后,z上下 右手 //Tango Device_Frame x左右,y上下,z前后 右手 //UE4 x前后,y左右,z上下 左手 static void TangoService_onPoseAvailable(void* context, const TangoPoseData* pose) { if (pose->status_code == TANGO_POSE_VALID && pose->frame.base == TANGO_COORDINATE_FRAME_START_OF_SERVICE && pose->frame.target == TANGO_COORDINATE_FRAME_DEVICE ) { translation[0] = pose->translation[0] * 100; translation[1] = pose->translation[1] * 100; translation[2] = pose->translation[2] * 100; quat.X = pose->orientation[0]; quat.Y = pose->orientation[1]; quat.Z = pose->orientation[2]; quat.W = pose->orientation[3]; } else { } } #endif }; "C++文件" #include "Office_05.h" #include "TangoApp.h" class UTextRenderComponent* TangoApp::textRender = nullptr; FVector TangoApp::translation = FVector::ZeroVector; FQuat TangoApp::quat = FQuat::Identity; TangoApp::TangoApp() { } TangoApp::~TangoApp() { }
在這里有個失敗的嘗試,我想通過NDK來申請ADF(區域文件相關權限),見上面的Launch方法,總是在調用CallVoidMethod時失敗,而上面的getPackageName又沒有問題,想不出來是啥問題。
通過工具Android Device Monitor,一般來說如果按照UE4的安卓工具包的流程來安裝,這個工具在目錄C:\NVPACK\android-sdk-windows\tools\monitor.bat下,打開Android Device Monitor,一般來說,我們新建一個Filters,如下圖設置。

如上圖設置后,我們得到錯誤信息是FindMethod得到的方法為空,到這一步后,相關參數應該沒有問題,可能是還要引入新的庫,總之,在這我們得不到區域文件,那么我們不能通過區域文件來定義,只能通過設備開始位置來定位了,這樣我們需要在特定的位置,特定的方向打開這個程序才能正確tracking現實與模型,這樣限制太大,所以我們需要提供一UI可以自己修改位置與方向,這樣,在開始時,我們先調到我們本身的位置與方向與項目的位置與方向重合。還好,這個東東並不需要我們多花費時間,UE4里本身的功能與內容包里,就有一個C++功能First Person,我們添加到項目中,這個在安卓下就提供二個圈給我們,一個圈調整水平位置,一個調整視角方向,剛好滿足我們的需求,現在我們結合First Person與Tango,讓Tango本身的路徑追蹤來替代First Person里的行走,如下是主要的修改位置。
void AFP_FirstPersonCharacter::BeginPlay() { Super::BeginPlay(); text = FindActor<ATextRenderActor>(TEXT("TextRenderActor2")); text1 = FindActor<ATextRenderActor>(TEXT("TextRenderActor3")); TangoApp::SetTextRender(text->GetTextRender()); #if PLATFORM_ANDROID TangoApp::InitTango(); //TangoApp::Launch(); #endif TangoApp::appendText(text->GetTextRender(), TEXT("T"), true); TangoApp::appendText(text1->GetTextRender(), TEXT("R"), true); } void AFP_FirstPersonCharacter::Tick(float DeltaTime) { Super::Tick(DeltaTime); #if PLATFORM_ANDROID //Tango Device_Frame/OpenGL x左右,y上下,z前后 右手 //UE4 x前后,y左右,z上下 左手 //Tango START_OF_SERVICE x左右,y前后,z上下 右手 //Unity x左右,y上下,z前后 左手 ////Device_Frame (Unreal camera to Drive) //FMatrix ucTd(-FVector::UpVector, FVector::ForwardVector, FVector::RightVector, FVector::ZeroVector); ////(Drive to START_SERVICE) //FTransform dTss(TangoApp::GetQuat(), TangoApp::GetTranslation(), FVector(1, 1, 1)); ////X,Y互換 (START_SERVICE to Unreal world) //FMatrix ssTuw(FVector::RightVector, -FVector::ForwardVector, FVector::UpVector, FVector::ZeroVector); ////ucTd * dTss * ssTuw //FTransform dTuw = FTransform(ucTd) * dTss * FTransform(ssTuw); //SetActorRelativeLocation(dTuw.GetLocation()); //auto rotator = dTuw.Rotator();// (dTuw.GetRotation() * FQuat::FQuat(FVector::ForwardVector, -PI / 2.0f)).Rotator(); //dTuw.Rotator();// auto ToConvert = TangoApp::GetQuat(); auto translation = TangoApp::GetTranslation(); FQuat TangoToUnrealQuat(ToConvert.Z, -ToConvert.X, -ToConvert.Y, ToConvert.W); //0.7071(rad/2) angle = 90 axis = (0,1,0) FQuat ConvertedQuat = (FQuat(0.0, 0.7071, 0.0, 0.7071) * TangoToUnrealQuat); if (WeaponRange < 0 || WeaponRange >100) { WeaponRange = 0; } FVector ConvertedFVector = 1 * cRotaror.RotateVector(FVector(translation[1], translation[0], translation[2] + WeaponRange)) + cPostion; SetActorRelativeLocation(ConvertedFVector); FRotator rotator = ConvertedQuat.Rotator() + cRotaror; if (Controller != nullptr) { Controller->SetControlRotation(rotator); } //if (GEngine) //{ // auto quat = ConvertedQuat; //dTuw.GetRotation();// // GEngine->AddOnScreenDebugMessage(0, 30.f, FColor::Red, "X:" + FString::SanitizeFloat(quat.X) // + " Y:" + FString::SanitizeFloat(quat.Y) // + " Z:" + FString::SanitizeFloat(quat.Z)); //} #endif if (GEngine) { GEngine->AddOnScreenDebugMessage(0, 30.f, FColor::Red, "H:" + FString::SanitizeFloat(WeaponRange)); } } void AFP_FirstPersonCharacter::MoveForward(float Value) { if (Value != 0.0f) { // Add movement in that direction AddMovementInput(GetActorForwardVector(), Value); cPostion += GetActorForwardVector()*Value; } } void AFP_FirstPersonCharacter::MoveRight(float Value) { if (Value != 0.0f) { // Add movement in that direction AddMovementInput(GetActorRightVector(), Value); cPostion += GetActorRightVector()*Value; } } void AFP_FirstPersonCharacter::TurnAtRate(float Rate) { // Calculate delta for this frame from the rate information AddControllerYawInput(Rate * BaseTurnRate * GetWorld()->GetDeltaSeconds()); cRotaror.Yaw += Rate * BaseTurnRate * GetWorld()->GetDeltaSeconds(); } void AFP_FirstPersonCharacter::LookUpAtRate(float Rate) { // Calculate delta for this frame from the rate information AddControllerPitchInput(Rate * BaseLookUpRate * GetWorld()->GetDeltaSeconds()); cRotaror.Pitch += Rate * BaseLookUpRate * GetWorld()->GetDeltaSeconds(); }
Tango里需要注意,每個攝像機,還有設備本身都采用不同的坐標系,如下圖。


想看更完全的介紹,請看 coordinate-systems 這個鏈接,如果不能打開,提示二個字,紅杏。
嗯,在這差不多了,一個簡單利用Tango tracking,漫游辦公室的小程序就有了,是不是有點AR的感覺。Tango這部分就差不多完了,但是有個小問題,UI能調整水平方向,高度不能調整,我們增加一個調整高度的UI,順便演練下藍圖調用C++的API的方法。
首先我們創建一個基於GameMode的我們自己的MyGameMode.

然后和上面一樣,創建一個基於HUD的自己的MyHUD,在MyGameMode中的HUD選擇MyHUD,在藍圖中,我們選擇添加用戶控件里的用戶藍圖,名字設為MyGUI.
在MyHUD中,添加MyGUI到視圖中,相應藍圖設置如下。

在MyGUI中添加一個Slider,其中OnValueChanged中設置如下:

其中FP_FirstPersonCharacter添加如下方法。
UFUNCTION(BlueprintCallable, Category = "Game") void SetHeight(float height);
特性聲明BlueprintCallable,其中Category是在藍圖中添加方法的分組名。
差不多完了,最后想起一個問題,在從 Unity 到 UE4 的快速上手與遷移 里的一個API,FindObject<T>,使用不成功,在AActor里使用反正得不到結果,同樣的參數,傳入如下函數就可以。
template< class T > T* FindActor(FString name) { for (TActorIterator<T> It(GetWorld()); It; ++It) { T* actor = *It; if (It->GetName() == name) { return actor; } } return nullptr; }
有知道的同學可以說下。
