ros圖像處理相關

1. image_transport

ros規定了多種基本的數據結構，用於node之間的傳輸。圖像也是一種常用的數據，image_transport package就是用來處理圖像數據傳輸的，它本身是個框架，只提供最基本的原始圖像數據(raw)，如果需要降低傳輸時的帶寬，還需要壓縮格式，由框架下集成的各種插件來完成。

image_transport 會發布 sensor_msgs/Image 格式的數據，其他壓縮格式由插件提供。

---

ROS中原始圖像格式"sensor_msgs/Image"如下：

                        # This message contains an uncompressed image
                        # (0, 0) is at top-left corner of image
                        #
Header header           # std_msgs/Header

uint32 height           # image height, that is, number of rows
uint32 width            # image width, that is, number of columns

                        # The legal values for encoding are in file src/image_encodings.cpp
                        # If you want to standardize a new string format, join
                        # [e-mail] and send an email proposing a new encoding.
string encoding         # Encoding of pixels -- channel meaning, ordering, size
                        # taken from the list of strings in include/sensor_msgs/image_encodings.h

uint8 is_bigendian      # is this data bigendian?
uint32 step             # Full row length in bytes
uint8[] data            # actual matrix data, size is (step * rows)

其中 header 是標准結構:

                        # Standard metadata for higher-level stamped data types.
                        # This is generally used to communicate timestamped data
                        # in a particular coordinate frame.
                        #
                        # sequence ID: consecutively increasing ID
uint32 seq

                        #Two-integer timestamp that is expressed as:
                        # * stamp.sec: seconds (stamp_secs) since epoch (in Python the variable is called 'secs')
                        # * stamp.nsec: nanoseconds since stamp_secs (in Python the variable is called 'nsecs')
                        # time-handling sugar is provided by the client library
time stamp

                        #Frame this data is associated with
string frame_id

一個具體例子:

root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw --noarr
header:
  seq: 293              # 當前幀序號
  stamp:                # 獲取圖像的時間戳
    secs: 1636868180
    nsecs:  53476744
  frame_id: "camera"    # frame_id
height: 480             # 行數
width: 640              # 列數
encoding: "bgr8"        # 像素點的壓縮格式，通道數3，順序是BGR，深度是24。按照這個可以知道data數組中值的意義
is_bigendian: 0         # 是否大端排序
step: 1920              # 步長，一行的字節數，寬度(640) x 像素點字節數(3) = 1920bytes。圖像總共 480行，也就是 1920x480=921600字節，也是下邊data數組長度
data: "<array type: uint8, length: 921600>"

---

使用上的例子
不使用 image_transport 的訂閱發布：

// Do not communicate images this way!
#include <ros/ros.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
  // ...
}

ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe("in_image_topic", 1, imageCallback);
ros::Publisher pub = nh.advertise<sensor_msgs::Image>("out_image_topic", 1);

使用 image_transport 的訂閱發布:

// Use the image_transport classes instead.
#include <ros/ros.h>
#include <image_transport/image_transport.h>

void imageCallback(const sensor_msgs::ImageConstPtr& msg)
{
  // ...
}

ros::NodeHandle nh;
image_transport::ImageTransport it(nh);  // 相當於將原始的nh封裝了一次
image_transport::Subscriber sub = it.subscribe("in_image_base_topic", 1, imageCallback);
image_transport::Publisher pub = it.advertise("out_image_base_topic", 1);

---

提供的東西:

API

publisher

image_transport::Publisher
image_transport本身只在基礎topic: <base topic> 上發布 sensor_msgs/Image 原始格式的圖像，如果有其他插件，會自動在基礎topic下的其他topic發布:<base topic>/<transport name>，比如基礎topic是 "/cv_camera",
image_transport 會發布 "/cv_camera/image_raw" 原始格式的topic，插件compressed_image_transport 會在 "/cv_camera/image_raw/compressed" 發布壓縮格式的圖像。

subscriber
image_transport::Subscriber
訂閱基礎topic后，會自動訂閱其他相關topic。

parameter

image_transport 本身不發布參數，但插件會發布各種參數，比如幀率、壓縮等級等，需要看插件本身。
發布者應該使用動態參數機制，讓參數更容易被訂閱者使用。
參數格式一般是: <base topic>/<transport name>/<parameter name>
比如: /cv_camera/image_raw/compressed/parameter_descriptions

node

$ rosrun image_transport republish [in_transport] in:=<in_base_topic> [out_transport] out:=<out_base_topic>

比如將 theora 格式的topic圖像解壓，重新發布到另一個topic，可以這樣寫:

$ rosrun image_transport republish theora in:=camera/image raw out:=camera/image_decompressed

如果一個node只發布原始的圖像格式，可以用 image_transport 重新發布成多種格式的topic。

命令行工具

$ rosrun image_transport list_transports

可以查看當前所有的package，看看是否有可用的圖像處理插件，以及插件狀態。

---

下邊的插件介紹
使用 image_transport 提供的一個node命令： list_transports，可以查看當前有的topic和對應的插件：

root@suntus:~# rosrun image_transport list_transports
Declared transports:
image_transport/compressed
image_transport/compressedDepth
image_transport/ffmpeg (*): Not available. Try 'catkin_make --pkg ffmpeg_image_transport'.
image_transport/raw
image_transport/theora

Details:
----------
"image_transport/compressed"
 - Provided by package: compressed_image_transport
 - Publisher:
      This plugin publishes a CompressedImage using either JPEG or PNG compression.

 - Subscriber:
      This plugin decompresses a CompressedImage topic.

----------
"image_transport/compressedDepth"
 - Provided by package: compressed_depth_image_transport
 - Publisher:
      This plugin publishes a compressed depth images using PNG compression.

 - Subscriber:
      This plugin decodes a compressed depth images.


----------
"image_transport/ffmpeg"
*** Plugins are not built. ***
 - Provided by package: ffmpeg_image_transport
 - Publisher:
      This plugin encodes frame into ffmpeg compressed packets

 - Subscriber:
      This plugin decodes frame from ffmpeg compressed packets

----------
"image_transport/raw"
 - Provided by package: image_transport
 - Publisher:
      This is the default publisher. It publishes the Image as-is on the base topic.

 - Subscriber:
      This is the default pass-through subscriber for topics of type sensor_msgs/Image.

----------
"image_transport/theora"
 - Provided by package: theora_image_transport
 - Publisher:
      This plugin publishes a video packet stream encoded using Theora.

 - Subscriber:
      This plugin decodes a video packet stream encoded using Theora.

---

compressed_image_transport("compressed"):
發布和訂閱 JPEG 或 PNG 壓縮格式的圖像，支持運行時改變圖片質量。

壓縮格式:

                    # This message contains a compressed image
Header header       # std_msgs/Header
string format       # Specifies the format of the data
                    #   Acceptable values:
                    #     jpeg, png
uint8[] data        # Compressed image buffer

一個例子:

root@suntus:~# rostopic echo -n 1 /cv_camera/image_raw/compressed --noarr
header:
  seq: 0
  stamp:
    secs: 1636880329
    nsecs: 751462995
  frame_id: "camera"
format: "bgr8; jpeg compressed bgr8"
data: "<array type: uint8, length: 43548>"
---

提供的東西：

發布插件

發布topic:

<base_topic>/compressed (sensor_msgs/CompressedImage)

參數

• <base_topic>/compressed/format (string, default: jpeg) Compression format to use, "jpeg" or "png".
• <base_topic>/compressed/jpeg_quality (int, default: 80) JPEG quality percentile, in the range [1, 100]. Lower values trade image quality for space savings.
• <base_topic>/compressed/png_level (int, default: 9) PNG compression level, in the range [1, 9]. Higher values trade computation time for space savings.

訂閱插件

訂閱topic:
<base_topic>/compressed (sensor_msgs/CompressedImage)

---

compressed_depth_image_transport
提供深度圖像(depth image),主要用於3D領域，帶距離的圖像格式。

---

theora_image_transport
發布成視頻流
theora視頻編碼格式的視頻流數據
theora是個視頻編碼格式，類似H264這種
對應的聲音編碼技術是vorbis
對應的文件封裝格式是ogg

發布topic
<base topic>/theora (theora_image_transport/Packet)

參數

• <base topic>/theora/optimize_for (int, default: Quality (1))
  Controls whether to use constant bitrate (CBR) encoding, aiming for ~theora/target_bitrate; or variable bitrate (VBR) encoding, aiming for ~theora/quality. Values are Bitrate (0) and Quality (1).
• <base topic>/theora/target_bitrate (int, default: 800000)
  Target bitrate. Used if optimize_for is set to Bitrate.
• <base topic>/theora/quality (int, default: 31)
  Encoding quality level, in the range [0, 63]. Used if optimize_for is set to Quality.
• <base topic>/theora/keyframe_frequency (int, default: 64)
  Maximum distance between key frames. If set to 1, every frame is a keyframe.

訂閱topic:
<base topic>/theora (theora_image_transport/Packet)

提供有一個node: ogg_saver，可以保存成 .ogv 格式的封裝文件，用其他播放器播放。

---

ffmpeg_image_transport
使用ffmpe將圖像轉換成h264或h265格式，並且支持nvidia GPU硬件加速。

https://github.com/daniilidis-group/ffmpeg_image_transport

---

cv_bridge
在ROS Image messages 和 OpenCV images 格式之間進行轉換。

ros圖像格式基礎的是 sensor_msgs/Image, 還有 sensor_msgs/CompressedImage, opencv中用到的格式為 cv::Mat 矩陣，需要進行轉換，才能放到opencv中使用。

opencv中支持的像素編碼格式有：
8UC[1-4]
8SC[1-4]
16UC[1-4]
16SC[1-4]
32SC[1-4]
32FC[1-4]
64FC[1-4]

cv_bridge有時候會進行必要的像素格式轉換，可以使用如下的字符串來表示目標格式：

• mono8: CV_8UC1, grayscale image
• mono16: CV_16UC1, 16-bit grayscale image
• bgr8: CV_8UC3, color image with blue-green-red color order
• rgb8: CV_8UC3, color image with red-green-blue color order
• bgra8: CV_8UC4, BGR color image with an alpha channel
• rgba8: CV_8UC4, RGB color image with an alpha channel

其中 mono8和bgr8是opencv中常用的格式。

---

roscpp

初始化和停止
ros::init(argc, argv, "my_node_name");
ros::shutdown();
ros::ok() 用於檢查node是否shutdown

timer

// 定時器，周期性執行回調。
ros::Timer ros::NodeHandle::createTimer(ros::Duration period, <callback>, bool oneshot = false);
ros::Timer timer = nh.createTimer(ros::Duration(0.1), timerCallback); // 創建timer

// 回調簽名:
void callback(const ros::TimerEvent&);
// 其中 ros::TimerEvent:
//      ros::Time last_expected     -- 理想中前一個回調應該被執行的時間
//      ros::Time last_real         -- 實際上前一個回調執行的時間
//      ros::Time current_expected  -- 理想中當前回調應該被執行的時間
//      ros::Time current_real      -- 實際上當前回調執行的時間
//      ros::WallTime profile.last_duration -- 前一個回調執行的耗時

線程

單線程

// 單線程調用
ros::init(argc, argv, "my_node");
ros::NodeHandle nh;
ros::Subscriber sub = nh.subscribe(...);
...
ros::spin();

// 自己實現 spin()
#include <ros/callback_queue.h>
ros::NodeHandle n;
while (ros::ok())
{
  ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0.1));
}


// 另一種形式的單線程調用
ros::Rate r(10); // 10 hz
while (should_continue)
{
  ... do some work, publish some messages, etc. ...
  ros::spinOnce();
  r.sleep();
}

// 自己實現 spinOnce()
#include <ros/callback_queue.h>
ros::getGlobalCallbackQueue()->callAvailable(ros::WallDuration(0));

多線程

// 同步spin，阻塞
ros::MultiThreadedSpinner spinner(4); // Use 4 threads
spinner.spin(); // spin() will not return until the node has been shutdown

// 異步spin
ros::AsyncSpinner spinner(4); // Use 4 threads
spinner.start();
ros::waitForShutdown(); // 這里會阻塞

調用隊列

// ros有個全局默認隊列, 所有回調都會掛到這個上邊
ros::getGlobalCallbackQueue()

// 創建自己的調用隊列，有兩層次:
#include <ros/callback_queue.h>
ros::CallbackQueue my_queue;

// 層次1: 每個 subscribe(), advertise(), advertiseService()
// 使用 Options 結構體傳入自己的隊列

// 層次2: 每個 NodeHandle，更常用. ros::spin() 不會自動觸發這些回調，需要手動去觸發
ros::NodeHandle nh;
nh.setCallbackQueue(&my_callback_queue);

// 一次調用所有的回調
callAvailable()

// 一次調用一個回調
callOne()