- NiViewer
安裝好攝像頭驅動和OpenNI后，在Tools文件夾中可以找到一個程序NiViewer。NiViewer的一些基本控制方法如下：
1. ESC關閉NiViewer程序
2. 右鍵可以顯示出控制選項
3. 按0到9以及“-”與“=”鍵，這12個鍵可以控制顯示的方法
4. 按M鍵控制顯示圖像是否鏡像
5. F鍵全屏切換
在使用NiViewer的時候，如果使用的設備沒有RGB攝像頭，此時NiViewer只顯示深度圖像，而RGB圖像部分則處於關閉狀態。如果使用的設備存在RGB攝像頭，則NiViewer窗口左半邊是3D攝像頭的深度影像，右半邊是RGB攝像頭的影像。
使用RGBD攝像頭時，由於需要傳輸的影像和深度信息資料的量很大，所以要注意所使用的USB的頻寬。當USB頻寬不夠的時候會出現卡頓或其他情況，盡量不要和其它大量數據傳輸的USB裝置同時使用。
- 深度感應器原理
對於光學設備來說，不可避免的要涉及精度和使用范圍問題。作為開發者，要熟悉這款設備的這些功能，對於基於這些設備所進行的開發有着決定性的作用。Xtion設備采用的是基於散斑的光源標定技術，所以散斑的形成與否對深度圖像有着決定性的作用。下圖是獲取的深度圖像，圖中黑色的部分表示無法精確獲得深度信息。首先在靠近鏡頭的地方會無法獲取深度，這是因為在這么近的距離無法形成有效的散斑。其次圖片中天花板上的照明燈管部分也無法獲取深度，因為Xtion使用的是主動光學式探測，所以直接光照會對傳感器的接收產生較強的干擾，最后一部分黑色是在圖片中最遠的地方，這是因為距離過遠，當距離過遠時也不能形成有效的散斑。
由於深度信息是通過紅外激光發射器發射激光，激光在物體表面通過漫反射形成散斑，再由CMOS傳感器接收返回的紅外激光來獲取，因此物體的材質會影響探測效果。比如透明材質和鏡面材質，對於這兩種材質，攝像頭均不能有效獲取深度信息。除了材質問題，在遮蔽的情況下，攝像頭也不能獲取到物體的深度。當一個物體被其他非透明物體所遮擋，這樣紅外激光不能照射到物體，也就不能被攝像頭的接收器所接收到，這時物體就沒有正確的深度。
- 獲取深度數據
具體創建工程的步驟可以參考OpenNI文檔中的Getting Started章節。下圖描述了數據獲取的流程：
1. 輪詢方式：
#include <stdio.h> #include <OpenNI.h> #include "OniSampleUtilities.h" #define SAMPLE_READ_WAIT_TIMEOUT 2000 //2000ms using namespace openni; // The entire C++ API is available under the openni namespace. int main(int argc, char* argv[]) { // Be sure to call openni::OpenNI::initialize(), to make sure all drivers are loaded // If no drivers are found, this function will fail. If it does, you can get some basic // log by calling openni::OpenNI::getExtendedError() (which returns a string) Status rc = OpenNI::initialize(); if (rc != STATUS_OK) { // getExtendedError() method returns additional, human-readable information about the error. printf("Initialize failed\n%s\n", OpenNI::getExtendedError()); return 1; } // Provides an interface to a single sensor device connected to the system. Requires OpenNI // to be initialized before it can be created. Devices provide access to Streams. Device device; // Device::open(): connects to a physical hardware device // This function returns a status code indicating either success or what error occurred. if (argc < 2) rc = device.open(ANY_DEVICE); else rc = device.open(argv[1]); if (rc != STATUS_OK) { printf("Couldn't open device\n%s\n", OpenNI::getExtendedError()); return 2; } VideoStream depth; // VideoStream: Abstracts a single video stream. Obtained from a specific Device. // Get the SensorInfo for a specific sensor type on this device // The SensorInfo is useful for determining which video modes are supported by the sensor // Parameters: sensorType of sensor to get information // Returns: SensorInfo object corresponding to the sensor type specified,or NULL if such a sensor is not available from this device. if (device.getSensorInfo(SENSOR_DEPTH) != NULL) { // Before VideoStream object can be used, this object must be initialized with the VideoStream::create() function. // The create() function requires a valid initialized device. Once created, // you should call the VideoStream::start() function to start the flow of data rc = depth.create(device, SENSOR_DEPTH); if (rc != STATUS_OK) { printf("Couldn't create depth stream\n%s\n", OpenNI::getExtendedError()); return 3; } } rc = depth.start(); // Start data generation from the video stream if (rc != STATUS_OK) { printf("Couldn't start the depth stream\n%s\n", OpenNI::getExtendedError()); return 4; } VideoFrameRef frame; // VideoFrameRef: Abstracts a single video from and related meta-data. Obtained from a specific Stream. // Polling based data reading while (!wasKeyboardHit()) { int changedStreamDummy; VideoStream* pStream = &depth; // A system of polling for stream access can be implemented by using the OpenNI::waitForAnyStream() function. // When called, it blocks until any of the streams in the list have new data available,or the timeout has passed. // It then returns a status code and indicates which stream has data available. rc = OpenNI::waitForAnyStream(&pStream, 1, &changedStreamDummy, SAMPLE_READ_WAIT_TIMEOUT); if (rc != STATUS_OK) { printf("Wait failed! (timeout is %d ms)\n%s\n", SAMPLE_READ_WAIT_TIMEOUT, OpenNI::getExtendedError()); continue; } // Once a VideoStream has been created, data can be read from it directly with the VideoStream::readFrame() function. rc = depth.readFrame(&frame); if (rc != STATUS_OK) { printf("Read failed!\n%s\n", OpenNI::getExtendedError()); continue; } // getVideoMode() can be used to determine the video mode settings of the sensor // This information includes the pixel format and resolution of the image, as well as the frame rate // PIXEL_FORMAT_DEPTH_1_MM: The values are in depth pixel with 1mm accuracy if (frame.getVideoMode().getPixelFormat() != PIXEL_FORMAT_DEPTH_1_MM && frame.getVideoMode().getPixelFormat() != PIXEL_FORMAT_DEPTH_100_UM) { printf("Unexpected frame format\n"); continue; } // VideoFrameRef::getData() function that returns a pointer directly to the underlying frame data. // This will be a void pointer, so it must be cast using the data type of the individual pixels in order to be properly indexed. DepthPixel* pDepth = (DepthPixel*)frame.getData(); // getHeight() and getWidth() functions are provided to easily determine the resolution of the frame int middleIndex = (frame.getHeight() + 1) * frame.getWidth() / 2; // getTimestamp: Provides timestamp of frame, measured in microseconds from an arbitrary zero printf("[%08llu] %8d\n", (long long)frame.getTimestamp(), pDepth[middleIndex]); // %md：m為指定的輸出字段的寬度。如果數據的位數小於m，則左端補以空格，若大於m，則按實際位數輸出 // 0：有0表示指定空位填0,如省略表示指定空位不填。 // u格式：以無符號十進制形式輸出整數。對長整型可以用"%lu"格式輸出 } depth.stop(); // Stop the flow of data depth.destroy(); // Destroy the stream // The close() function properly shuts down the hardware device. As a best practice, any device // that is opened should be closed. This will leave the driver and hardware device in a known // state so that future applications will not have difficulty connecting to them. device.close(); // When an application is ready to exit, the OpenNI::shutdown() function // should be called to shutdown all drivers and properly clean up. OpenNI::shutdown(); return 0; }
2. 事件方式：
#include <stdio.h> #include "OpenNI.h" #include "OniSampleUtilities.h" using namespace openni; // VideoFrameRef class encapsulates all data related to a single frame read from a VideoStream void analyzeFrame(const VideoFrameRef& frame) { DepthPixel* pDepth; RGB888Pixel* pColor; // This structure stores a single color pixel value(in 24-bit RGB format). int middleIndex = (frame.getHeight()+1)*frame.getWidth()/2; // getVideoMode() can be used to determine the video mode settings of the sensor // This information includes the pixel format and resolution of the image, as well as the frame rate switch (frame.getVideoMode().getPixelFormat()) { case PIXEL_FORMAT_DEPTH_1_MM: case PIXEL_FORMAT_DEPTH_100_UM: // VideoFrameRef::getData() returns a pointer directly to the underlying frame data. pDepth = (DepthPixel*)frame.getData(); printf("[%08llu] %8d\n", (long long)frame.getTimestamp(), pDepth[middleIndex]); break; case PIXEL_FORMAT_RGB888: pColor = (RGB888Pixel*)frame.getData(); printf("[%08llu] 0x%02x%02x%02x\n", (long long)frame.getTimestamp(), pColor[middleIndex].r&0xff, pColor[middleIndex].g&0xff, pColor[middleIndex].b&0xff); break; default: printf("Unknown format\n"); } } // The VideoStream::NewFrameListener class is provided to allow the implementation of event driven frame reading. // To use it, create a class that inherits from it and implement override the onNewFrame() method class PrintCallback : public VideoStream::NewFrameListener { public: void onNewFrame(VideoStream& stream) { // Once a VideoStream has been created, data can be read from it directly with the VideoStream::readFrame() function. stream.readFrame(&m_frame); // VideoFrameRef objects are obtained from calling VideoStream::readFrame() analyzeFrame(m_frame); } private: VideoFrameRef m_frame; }; class OpenNIDeviceListener : public OpenNI::DeviceConnectedListener, public OpenNI::DeviceDisconnectedListener, public OpenNI::DeviceStateChangedListener { public: // All three events provide a pointer to a openNI::DeviceInfo object. This object can be used to get // details and identification of the device referred to by the event virtual void onDeviceStateChanged(const DeviceInfo* pInfo, DeviceState state) { printf("Device \"%s\" error state changed to %d\n", pInfo->getUri(), state); } virtual void onDeviceConnected(const DeviceInfo* pInfo) { printf("Device \"%s\" connected\n", pInfo->getUri()); } virtual void onDeviceDisconnected(const DeviceInfo* pInfo) { printf("Device \"%s\" disconnected\n", pInfo->getUri()); } }; int main() { Status rc = OpenNI::initialize(); if (rc != STATUS_OK) { printf("Initialize failed\n%s\n", OpenNI::getExtendedError()); return 1; } OpenNIDeviceListener devicePrinter; // OpenNI defines three events: onDeviceConnected, onDeviceDisconnected, and onDeviceStateChanged // Listener classes can be added or removed from the list of event handlers OpenNI::addDeviceConnectedListener(&devicePrinter); OpenNI::addDeviceDisconnectedListener(&devicePrinter); OpenNI::addDeviceStateChangedListener(&devicePrinter); openni::Array<openni::DeviceInfo> deviceList; // OpenNI::enumerateDevices() returns a list of all available devices connected to the system. openni::OpenNI::enumerateDevices(&deviceList); for (int i = 0; i < deviceList.getSize(); ++i) { // DeviceInfo::getUri returns te divice URI. URI can be used by Device::open to open a specific device. printf("Device \"%s\" already connected\n", deviceList[i].getUri()); } Device device; rc = device.open(ANY_DEVICE); if (rc != STATUS_OK) { printf("Couldn't open device\n%s\n", OpenNI::getExtendedError()); return 2; } VideoStream depth; if (device.getSensorInfo(SENSOR_DEPTH) != NULL) { rc = depth.create(device, SENSOR_DEPTH); if (rc != STATUS_OK) { printf("Couldn't create depth stream\n%s\n", OpenNI::getExtendedError()); } } rc = depth.start(); if (rc != STATUS_OK) { printf("Couldn't start the depth stream\n%s\n", OpenNI::getExtendedError()); } PrintCallback depthPrinter; // Register your created class with an active VideoStream using the VideoStream::addNewFrameListener() function. // Once this is done, the event handler function you implemented will be called whenever a new frame becomes available. depth.addNewFrameListener(&depthPrinter); // Wait while we're getting frames through the printer while (!wasKeyboardHit()) { Sleep(100); } // Removes a Listener from this video stream list. The listener removed will no longer receive new frame events from this stream depth.removeNewFrameListener(&depthPrinter); depth.stop(); depth.destroy(); device.close(); OpenNI::shutdown(); return 0; }
深度圖像中每一個像素點所存儲的數據就是這個像素點距離攝像頭的距離，也就是深度，它可以看做一張由深度像素組成的一維矩陣，其大小為nXRes*nYRes（即X軸像素*Y軸像素）。如果輸出模式為320*240、30幀的格式，當深度數據流開始時，就是以每秒30幀的速率產生一張像素為320*240的圖。
代碼中聲明了一個DepthPixel類型的指針pDepth，實質是一個用於指向存放深度圖的一維數組。則圖像中心點的深度信息middleIndex可以由下式算出：
// getHeight() and getWidth() functions are provided to easily determine the resolution of the frame int middleIndex = (frame.getHeight() + 1) * frame.getWidth() / 2;
這里需要注意，深度圖中像素值直接代表深度，單位是mm。在通過convertDepthToWorld這個函數轉換到真實坐標系（World coordinates）時Z坐標是不變的。函數會根據Depth坐標系下的(X, Y, Z)，去計算World坐標系下的位置(X’, Y’, Z)，X、Y是depth map的索引位置（X：為column ，Y：為row）。也就是說，在OpenNI里Depth坐標系和World坐標系下Z坐標的意義是相同的。所以，如果只想知道某個點的深度，其實不用轉換到真實坐標系中，直接在投影坐標系對Z值做計算就可以了。但是如果想知道兩點的距離，那就還是要進行轉換，把真實坐標系的X、Y坐標計算出來。
OpenNI applications commonly use two different coordinate systems to represent depth. These two systems are referred to as Depth and World representation. Depth coordinates are the native data representation. In this system, the frame is a map (two dimensional array), and each pixel is assigned a depth value. This depth value represents the distance between the camera plane and whatever object is in the given pixel. The X and Y coordinates are simply the location in the map, where the origin is the top-left corner of the field of view.
World coordinates superimpose a more familiar 3D Cartesian coordinate system on the world, with the camera lens at the origin. In this system, every point is specified by 3 points – x, y and z. The x axis of this system is along a line that passes through the infrared projector and CMOS imager of the camera. The y axis is parallel to the front face of the camera, and perpendicular to the x axis (it will also be perpendicular to the ground if the camera is upright and level). The z axis runs into the scene, perpendicular to both the x and y axis. From the perspective of the camera, an object moving from left to right is moving along the increasing x axis. An object moving up is moving along the increasing y axis, and an object moving away from the camera is moving along the increasing z axis.
Mathematically, the Depth coordinate system is the projection of the scene on the CMOS. If the sensor's angular field of view and resolution are known, then an angular size can be calculated for each pixel. This is how the conversion algorithms work.
Note that converting from Depth to World coordinates is relatively expensive computationally. It is generally not practical to convert the entire raw depth map to World coordinates. A better approach is to have your computer vision algorithm work in Depth coordinates for as long as possible, and only converting a few specific points to World coordinates right before output. Note that when converting from Depth to World or vice versa, the Z value remains the same.
參考：
《OpenNI體感應用開發實戰》 機械工業出版社