1.前言
本文介紹了webrtc中的fec相關封裝原理, 協議，分析其在webrtc中的應用過程和使用策略。
2.正文
2.1 red
為什么做red封裝呢？Ulpfec編碼后的內容會做Red封裝后再放入RtpPacket，可fec在RFC5109已經定義好自己的傳輸格式，而且sdp協商過程中也有Ulpfec的PT為a=rtpmap:125 ulpfec/90000, 可以通過RtpHeader的PT進行區分出Ulpfec包，看起來是多此一舉. 其實不然，fec過程中要傳輸的包有兩種，一種是冗余包，一種是原始包，將他們統一的放在red中，表示這些包屬於一個fec流，然后red提供的pt可以分辨出那些是fec包，哪些是原始數據包
2.1.1 red封裝格式
Red的封裝如下,由block header + data block組成，payload會被放在data block中
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| block PT | timestamp offset | block length | : block header
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| block | : data block
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
介紹一下各個標志位
F: 為1時標識后面是否有別的block,如果有則會出現多個這樣的header line
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| block PT | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|F| block PT | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.
.
.
為0時，標識后面不會再有這個header line，同時最后block的timestamp和block
length能夠通過rtp推斷出，所以會省略:
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|1| block PT=7 | timestamp offset | block length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|0| block PT=5 | |
+-+-+-+-+-+-+-+-+
block PT: block內容中的rtp payload type
block length: block的大小
timestamp offset: block 的timestam 相對於rtp header timestamp 的offset
一個完整的red如下所示：
// 0 1 2 3
// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |V=2|P|X| CC=0 |M| PT | sequence number of primary |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// | timestamp of primary encoding |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// | synchronization source (SSRC) identifier |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |1| block PT=7 | timestamp offset | block length |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
// |0| block PT=5 | |
// +-+-+-+-+-+-+-+-+ +
// | |
// + LPC encoded redundant data (PT=7) +
// | (14 bytes) |
// + +---------------+
// | | |
// +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +
// | |
// + +
// | |
// + +
// | |
// + +
// | DVI4 encoded primary data (PT=5) |
// + (84 bytes, not to scale) +
// / /
// + +
// | |
// + +
// | |
// + +---------------+
// | |
由於webrtc只用到了單個block的red封裝，就是在payload前加多1Byte，把payloadtype放進去
2.1.2 red封裝流程
RTPSenderVideo::SendVideo()是對packet進行RTP打包的地方，會檢查是否開啟了red封裝，會將打好的包重新設置成red_packet
bool RTPSenderVideo::SendVideo(
int payload_type,
absl::optional<VideoCodecType> codec_type,
uint32_t rtp_timestamp,
int64_t capture_time_ms,
rtc::ArrayView<const uint8_t> payload,
RTPVideoHeader video_header,
absl::optional<int64_t> expected_retransmission_time_ms,
absl::optional<int64_t> estimated_capture_clock_offset_ms) {
if (red_enabled()) {
// 將packet替換成red_packet，將media payload放進red_packet中
// TODO(sprang): Consider packetizing directly into packets with the RED
// header already in place, to avoid this copy.
std::unique_ptr<RtpPacketToSend> red_packet(new RtpPacketToSend(*packet));
BuildRedPayload(*packet, red_packet.get()); // 生成red包
red_packet->SetPayloadType(*red_payload_type_);// 設定payloadType為Red
red_packet->set_is_red(true);// 標識此包為一個red包
// Append |red_packet| instead of |packet| to output.
red_packet->set_packet_type(RtpPacketMediaType::kVideo);
red_packet->set_allow_retransmission(packet->allow_retransmission());
rtp_packets.emplace_back(std::move(red_packet));
} else {
packet->set_packet_type(RtpPacketMediaType::kVideo);
rtp_packets.emplace_back(std::move(packet));
}
....
}
這里首先會受到enable red的影響，去標識當前包要做red封包(rfc2189), 其中：
-
red_packet本身的類型和普通包的類型都是RtpPacketToSend
-
調用
BuildRedPayload()去做Red封裝, 首先在此介紹一下Red封裝void BuildRedPayload(const RtpPacketToSend& media_packet, RtpPacketToSend* red_packet) { // 新增1Byte，用於放Red Header uint8_t* red_payload = red_packet->AllocatePayload( kRedForFecHeaderLength + media_packet.payload_size()); RTC_DCHECK(red_payload); red_payload[0] = media_packet.PayloadType(); //填入payload type // 拷貝payload 到block上 auto media_payload = media_packet.payload(); memcpy(&red_payload[kRedForFecHeaderLength], media_payload.data(), media_payload.size()); } -
通過red_packet->set_is_red(true)表示了這是一個red封裝的包
-
通過red_packet->SetPayloadType(*red_payload_type_) 設置了當前包的payloadType為red_payload_type，表征了payload為red，這個值一般是127, 這個config來自於struct UlpfecConfig
struct UlpfecConfig {
UlpfecConfig()
: ulpfec_payload_type(-1),
red_payload_type(-1),
red_rtx_payload_type(-1) {}
std::string ToString() const;
bool operator==(const UlpfecConfig& other) const;
// Payload type used for ULPFEC packets.
int ulpfec_payload_type;
// Payload type used for RED packets.
int red_payload_type;
// RTX payload type for RED payload.
int red_rtx_payload_type;
};
2.2 fec
2.2.1 ulpfec原理
fec的原理很簡單大家都知道，就是異或，比如說傳輸兩個包 P1 和 P2, 通過 異或P1，P2得到FEC包 F1, 那么傳輸過程中能收到任意兩個，都可以通過異或恢復出原來的 P1 和 P2。
兩個包的問題好考慮，但是多個包的時候會引入第二個問題，為了帶寬控制fec包的數量時候，應該選擇哪一部分包作為一個集合去異或出一個fec包呢? 比如有10個數據包，當前一段傳輸一段時間內丟包率為30%，此時可以引入一個簡單的處理策略: 將fec包的數量控制為丟包率也就是3個，但這3個FEC包由哪一塊做異或呢？ 如下所示，有兩種分配方式，一種是以相鄰的包為一組做fec，這時候每3個包可能丟一個，不影響恢復，這種模型能較好的對抗隨機丟包的網絡，第二種則是取等間隔的包做一組，突發性丟包會導致連續的一塊包丟失，這種模型對突發丟包的網絡比較好
2.2.2 ulpfec封裝格式
詳見RFC5109, 在Rtp傳輸時上，跟在RTPheader后，可以理解作為payload吧
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| RTP Header (12 octets or more) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Header (10 octets) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 0 Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 0 Payload |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 1 Header |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FEC Level 1 Payload |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Cont. |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
其中 FEC Header如下所示:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|E|L|P|X| CC |M| PT recovery | SN base |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| TS recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| length recovery |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
E: 標識是否啟用擴展頭，當前值為0，保留位，
L: 使用的mask表的長度，0-->16bits, 1-->48bits
P, X, CC, M, PT recovery: 進行異或的Rtp包頭上的P, X, CC, M, PT的異或結果
SN base: 進行異或的Rtp包的最小sequence號
Ts recovery: 進行異或的Rtp包的Timestamp的異或結果
length: 進行異或的Rtp包的payload長度的異或結果
ulpfec中可以將不同的數據使用不同的level級別去做保護, 如下:
Packet A #####################
: :
Packet B ############### :
: :
ULP FEC Packet #1 @@@@@@@@ :
: :
Packet C ########### :
: :
Packet D ###################################
: :
ULP FEC Packet #2 @@@@@@@@@@@@@@@@@
: : :
:<-L0->:<--L1-->:
Payload packet # | ULP FEC packet that protects at level
| L0 L1
---------------------+---------------------------------------
A | #1 #2
B | #1 #2
C | #2 #2
D | #2 #2
對A和B的前半段數據使用#1包去保護，對C和D的前半段數據用#2包去保護，這前半段數據都處於L0級別
對於A,B,C,D的后半段數據使用#2包保護，這些保護的數據被設在L1級別
這里認為A,B,C,D的后半段的數據沒那么重要，用4:1的比例做冗余就夠了，而前半段的數據很重要，所以需要2:1的比例做冗余(一直覺得RFC上這圖畫的是不是有問題，L1的長度這么短，后面的數據是不要了嗎，嘻)；
因為一個FEC包中可能有不同level級別的數據，為此在Fec header中引入了 FEC Level 0 Header 和 FEC Level 1 Header，結構如下所示，主要是標記了保護的數據長度和參與fec包的掩碼表
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Protection Length | mask |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| mask cont. (present only when L = 1) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Protection Length： L0包保護的長度
mask: 標記packet是否參與異或的掩碼表，每個bit標記着以fec header的SN base作為偏移起始的rtp Sequence對應的包是否參與當前fec block的編碼
mask cont：當L被設置成1的時候 mask cont會出現和mask連起來形成48bit的掩碼表
但這里需要明確一下，webrtc中沒有用到L1這個級別，因為packet中的信息都是幀的一部分優先級都是一樣的，但並不是說L1這個東西不能用，可以把上面的粒度替換一下，packet換成Frame，后面的數據換成包也可以嘗試一番。
2.2.3 ulpfec編碼過程
2.2.3.1 解red封裝
webrtc做fec冗余的地方在隨后發送鏈路中的RtpSenderEgress::SendPacket()
void RtpSenderEgress::SendPacket(RtpPacketToSend* packet,
const PacedPacketInfo& pacing_info) {
....
//fec處理
if (fec_generator_ && packet->fec_protect_packet()) {
// This packet should be protected by FEC, add it to packet generator.
RTC_DCHECK(fec_generator_);
RTC_DCHECK(packet->packet_type() == RtpPacketMediaType::kVideo);
absl::optional<std::pair<FecProtectionParams, FecProtectionParams>>
new_fec_params;
{
MutexLock lock(&lock_);
new_fec_params.swap(pending_fec_params_);
}
// fec_rate和fec_max_frame 可能被更新了
if (new_fec_params) {
fec_generator_->SetProtectionParameters(new_fec_params->first,
new_fec_params->second);
}
if (packet->is_red()) {
// 對packet做了red封裝(rfc2198)，需要解red封裝得到原始rtp包
// 很奇怪，會進行red封包的應該只有fec,普通包也會進行fec封裝？
// 復制整個包
RtpPacketToSend unpacked_packet(*packet);
const rtc::CopyOnWriteBuffer buffer = packet->Buffer();
// Grab media payload type from RED header.
const size_t headers_size = packet->headers_size();
unpacked_packet.SetPayloadType(buffer[headers_size]);
// 對copyonwirte的payload進行拷貝
// Copy the media payload into the unpacked buffer.
uint8_t* payload_buffer =
unpacked_packet.SetPayloadSize(packet->payload_size() - 1);
std::copy(&packet->payload()[0] + 1,
&packet->payload()[0] + packet->payload_size(), payload_buffer);
// 對包做fec
fec_generator_->AddPacketAndGenerateFec(unpacked_packet);
} else {
// If not RED encapsulated - we can just insert packet directly.
fec_generator_->AddPacketAndGenerateFec(*packet);
}
}
.....
}
主要做了兩件事:
- 通過packet->fec_protect_packet()檢查packet是否啟用了fec，是則使用fec_generator_->AddPacketAndGenerateFec()將包加入到fec的隊列中准備做fec
- 如果packet做了red封裝，需要對payload解red封裝，也就是那個1 Byte的red header給去掉
2.2.3.2 更新fec params
此處用的是ulpfec，接下來的函數是UlpfecGenerator::AddPacketAndGenerateFec()
void UlpfecGenerator::AddPacketAndGenerateFec(const RtpPacketToSend& packet) {
RTC_DCHECK_RUNS_SERIALIZED(&race_checker_);
RTC_DCHECK(generated_fec_packets_.empty());
{
MutexLock lock(&mutex_);
// 更新fec params
if (pending_params_) {
current_params_ = *pending_params_;
pending_params_.reset();
// 重設的閾值高於kHighProtectionThreshold, 更改最少需要4個包去做fec encode
if (CurrentParams().fec_rate > kHighProtectionThreshold) {
min_num_media_packets_ = kMinMediaPackets;
} else {
min_num_media_packets_ = 1;
}
}
}
if (packet.is_key_frame()) {
media_contains_keyframe_ = true;
}
const bool complete_frame = packet.Marker();
if (media_packets_.size() < kUlpfecMaxMediaPackets) {
// 構造fec packet放入等待隊列中
// Our packet masks can only protect up to |kUlpfecMaxMediaPackets| packets.
auto fec_packet = std::make_unique<ForwardErrorCorrection::Packet>();
fec_packet->data = packet.Buffer();
media_packets_.push_back(std::move(fec_packet));
// Keep a copy of the last RTP packet, so we can copy the RTP header
// from it when creating newly generated ULPFEC+RED packets.
RTC_DCHECK_GE(packet.headers_size(), kRtpHeaderSize);
last_media_packet_ = packet;
}
if (complete_frame) {
++num_protected_frames_;
}
auto params = CurrentParams();
// Produce FEC over at most |params_.max_fec_frames| frames, or as soon as:
// (1) the excess overhead (actual overhead - requested/target overhead) is
// less than |kMaxExcessOverhead|, and
// (2) at least |min_num_media_packets_| media packets is reached.
if (complete_frame &&
(num_protected_frames_ >= params.max_fec_frames ||
(ExcessOverheadBelowMax() && MinimumMediaPacketsReached()))) {
// We are not using Unequal Protection feature of the parity erasure code.
constexpr int kNumImportantPackets = 0;
constexpr bool kUseUnequalProtection = false;
// fec編碼
fec_->EncodeFec(media_packets_, params.fec_rate, kNumImportantPackets,
kUseUnequalProtection, params.fec_mask_type,
&generated_fec_packets_);
if (generated_fec_packets_.empty()) {
ResetState();
}
}
}
主要做了:
-
更新ulpfec的params，結構如下，對於關鍵幀和P幀各有一套獨立的控制參數，控制參數主要有
fec_rate: 為(fec包數)/(媒體包數) * 256，調整策略見 2.3
max_fec_frames: fec編碼是針對收集到的一段packet進行fec編碼的，此參數規定到了這個數量時一定要fec編碼了
fec_mask_type：webrtc根據丟包模型准備了兩個fec的掩碼表，隨機丟包掩碼表和突發丟包掩碼表
struct Params { Params(); Params(FecProtectionParams delta_params, FecProtectionParams keyframe_params); FecProtectionParams delta_params; // p幀 FecProtectionParams keyframe_params;// 關鍵幀 }; struct FecProtectionParams { int fec_rate = 0; // fec_packet_num / media_packet_num * 256 int max_fec_frames = 0; // 經過max_fec_frames一定要生成fec包 FecMaskType fec_mask_type = FecMaskType::kFecMaskRandom;// 選擇使用給的fec掩碼表 }; -
當fec_rate大於kHighProtectionThreshold(80)時，重設了最小FEC編碼packet數為4，不太清楚這樣改動的意義，是怕太頻繁重復encode嗎
-
構建Fec packet，將payload拷貝進去
-
檢查是否立即進行FEC encode，滿足的條件有一下兩種，滿足任意一個就調用fec_->EncodeFec()進行編碼
1.完整幀 + 收集的幀數量超過設定值
2.完整幀 + 達到設定最小收集包數 + 當前做FEC所得到的rate和預設的fec_rate相減小於一個閾值
關於加黑的條件也就是ExcessOverheadBelowMax()，有一些細節, 在實際計算要用到的fec包的數量時，在fec_rate的基礎上還加了個0.5的上取整，這是導致實際和預設出現差值的原因
// 通過fec_rate計算實際需要用到的fec包數 int ForwardErrorCorrection::NumFecPackets(int num_media_packets, int protection_factor) { // Result in Q0 with an unsigned round. // fec = media * fec_rate / 256 + 0.5 此處做了個上取整 int num_fec_packets = (num_media_packets * protection_factor + (1 << 7)) >> 8; // Generate at least one FEC packet if we need protection. if (protection_factor > 0 && num_fec_packets == 0) { num_fec_packets = 1; } RTC_DCHECK_LE(num_fec_packets, num_media_packets); return num_fec_packets; } int UlpfecGenerator::Overhead() const { RTC_DCHECK_RUNS_SERIALIZED(&race_checker_); RTC_DCHECK(!media_packets_.empty()); int num_fec_packets = fec_->NumFecPackets(media_packets_.size(), CurrentParams().fec_rate); // overheade的計算，用實際需要使用的 fec包數 * 256 / media包數 return (num_fec_packets << 8) / media_packets_.size(); } bool UlpfecGenerator::ExcessOverheadBelowMax() const { RTC_DCHECK_RUNS_SERIALIZED(&race_checker_); // 實際 - 預設 < 50 return ((Overhead() - CurrentParams().fec_rate) < kMaxExcessOverhead); }同時，注意到注釋中說沒有使用Unequal Protection feature of the parity erasure code.沒有啟動L1級別做FEC；
2.2.3.3 開始Encodefec
接下來到了實際的fec編碼過程ForwardErrorCorrection::EncodeFec()
int ForwardErrorCorrection::EncodeFec(const PacketList& media_packets,
uint8_t protection_factor,
int num_important_packets,
bool use_unequal_protection,
FecMaskType fec_mask_type,
std::list<Packet*>* fec_packets) {
const size_t num_media_packets = media_packets.size();
// Sanity check arguments.
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GE(num_important_packets, 0);
RTC_DCHECK_LE(num_important_packets, num_media_packets);
RTC_DCHECK(fec_packets->empty());
const size_t max_media_packets = fec_header_writer_->MaxMediaPackets();
if (num_media_packets > max_media_packets) {
RTC_LOG(LS_WARNING) << "Can't protect " << num_media_packets
<< " media packets per frame. Max is "
<< max_media_packets << ".";
return -1;
}
// Error check the media packets.
for (const auto& media_packet : media_packets) {
RTC_DCHECK(media_packet);
if (media_packet->data.size() < kRtpHeaderSize) {
RTC_LOG(LS_WARNING) << "Media packet " << media_packet->data.size()
<< " bytes "
"is smaller than RTP header.";
return -1;
}
// Ensure the FEC packets will fit in a typical MTU.
if (media_packet->data.size() + MaxPacketOverhead() + kTransportOverhead >
IP_PACKET_SIZE) {
RTC_LOG(LS_WARNING) << "Media packet " << media_packet->data.size()
<< " bytes "
"with overhead is larger than "
<< IP_PACKET_SIZE << " bytes.";
}
}
// Prepare generated FEC packets.
int num_fec_packets = NumFecPackets(num_media_packets, protection_factor);
if (num_fec_packets == 0) {
return 0;
}
for (int i = 0; i < num_fec_packets; ++i) {
generated_fec_packets_[i].data.EnsureCapacity(IP_PACKET_SIZE);
memset(generated_fec_packets_[i].data.MutableData(), 0, IP_PACKET_SIZE);
// Use this as a marker for untouched packets.
generated_fec_packets_[i].data.SetSize(0);
fec_packets->push_back(&generated_fec_packets_[i]);
}
// 根據丟包模型選擇mask表，生成mask
internal::PacketMaskTable mask_table(fec_mask_type, num_media_packets);
packet_mask_size_ = internal::PacketMaskSize(num_media_packets);
memset(packet_masks_, 0, num_fec_packets * packet_mask_size_);
// 生成mask表
internal::GeneratePacketMasks(num_media_packets, num_fec_packets,
num_important_packets, use_unequal_protection,
&mask_table, packet_masks_);
// Adapt packet masks to missing media packets.
int num_mask_bits = InsertZerosInPacketMasks(media_packets, num_fec_packets);
if (num_mask_bits < 0) {
RTC_LOG(LS_INFO) << "Due to sequence number gaps, cannot protect media "
"packets with a single block of FEC packets.";
fec_packets->clear();
return -1;
}
packet_mask_size_ = internal::PacketMaskSize(num_mask_bits);
// 開始FEC
// Write FEC packets to |generated_fec_packets_|.
GenerateFecPayloads(media_packets, num_fec_packets);
// TODO(brandtr): Generalize this when multistream protection support is
// added.
const uint32_t media_ssrc = ParseSsrc(media_packets.front()->data.data());
const uint16_t seq_num_base =
ParseSequenceNumber(media_packets.front()->data.data());
FinalizeFecHeaders(num_fec_packets, media_ssrc, seq_num_base);
return 0;
}
ForwardErrorCorrection::EncodeFec()主要:
- 如2.2.1中所介紹的，隨機丟包和突發丟包下fec組包模式不同，所以webrtc准備了兩張mask表 kFecMaskRandom(隨機丟包), kFecMaskBursty(突發丟包)去生成mask, 調用internal::GeneratePacketMasks()去生成mask
- 根據mask和packet, 調用GenerateFecPayloads()生成fec包
- 調用FinalizeFecHeaders()填入ssrc
2.2.3.4 生成mask
以下為調用要生成mask的internal::GeneratePacketMasks()中
void GeneratePacketMasks(int num_media_packets,
int num_fec_packets,
int num_imp_packets,
bool use_unequal_protection,
PacketMaskTable* mask_table,
uint8_t* packet_mask) {
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GT(num_fec_packets, 0);
RTC_DCHECK_LE(num_fec_packets, num_media_packets);
RTC_DCHECK_LE(num_imp_packets, num_media_packets);
RTC_DCHECK_GE(num_imp_packets, 0);
const int num_mask_bytes = PacketMaskSize(num_media_packets);
// Equal-protection for these cases.
if (!use_unequal_protection || num_imp_packets == 0) {
// Retrieve corresponding mask table directly:for equal-protection case.
// Mask = (k,n-k), with protection factor = (n-k)/k,
// where k = num_media_packets, n=total#packets, (n-k)=num_fec_packets.
// 調用PacketMaskTable::LookUp
rtc::ArrayView<const uint8_t> mask =
mask_table->LookUp(num_media_packets, num_fec_packets);
memcpy(packet_mask, &mask[0], mask.size());
} else { // UEP case
UnequalProtectionMask(num_media_packets, num_fec_packets, num_imp_packets,
num_mask_bytes, packet_mask, mask_table);
} // End of UEP modification
} // End of GetPacketMasks
由於沒有使用unequal protection，所以直接進入PacketMaskTable::LookUp()
PacketMaskTable::PacketMaskTable(FecMaskType fec_mask_type,
int num_media_packets)
: table_(PickTable(fec_mask_type, num_media_packets)) {}
PacketMaskTable::~PacketMaskTable() = default;
rtc::ArrayView<const uint8_t> PacketMaskTable::LookUp(int num_media_packets,
int num_fec_packets) {
RTC_DCHECK_GT(num_media_packets, 0);
RTC_DCHECK_GT(num_fec_packets, 0);
RTC_DCHECK_LE(num_media_packets, kUlpfecMaxMediaPackets);
RTC_DCHECK_LE(num_fec_packets, num_media_packets);
if (num_media_packets <= 12) {
// 小於12直接查表
return LookUpInFecTable(table_, num_media_packets - 1, num_fec_packets - 1);
}
int mask_length =
static_cast<int>(PacketMaskSize(static_cast<size_t>(num_media_packets)));
// Generate FEC code mask for {num_media_packets(M), num_fec_packets(N)} (use
// N FEC packets to protect M media packets) In the mask, each FEC packet
// occupies one row, each bit / coloumn represent one media packet. E.g. Row
// A, Col/Bit B is set to 1, means FEC packet A will have protection for media
// packet B.
// 大於12，使用interleaved，也就是間隔包為一組(X % N)
// Loop through each fec packet.
for (int row = 0; row < num_fec_packets; row++) {
// Loop through each fec code in a row, one code has 8 bits.
// Bit X will be set to 1 if media packet X shall be protected by current
// FEC packet. In this implementation, the protection is interleaved, thus
// media packet X will be protected by FEC packet (X % N)
for (int col = 0; col < mask_length; col++) {
fec_packet_mask_[row * mask_length + col] =
((col * 8) % num_fec_packets == row && (col * 8) < num_media_packets
? 0x80
: 0x00) |
((col * 8 + 1) % num_fec_packets == row &&
(col * 8 + 1) < num_media_packets
? 0x40
: 0x00) |
((col * 8 + 2) % num_fec_packets == row &&
(col * 8 + 2) < num_media_packets
? 0x20
: 0x00) |
((col * 8 + 3) % num_fec_packets == row &&
(col * 8 + 3) < num_media_packets
? 0x10
: 0x00) |
((col * 8 + 4) % num_fec_packets == row &&
(col * 8 + 4) < num_media_packets
? 0x08
: 0x00) |
((col * 8 + 5) % num_fec_packets == row &&
(col * 8 + 5) < num_media_packets
? 0x04
: 0x00) |
((col * 8 + 6) % num_fec_packets == row &&
(col * 8 + 6) < num_media_packets
? 0x02
: 0x00) |
((col * 8 + 7) % num_fec_packets == row &&
(col * 8 + 7) < num_media_packets
? 0x01
: 0x00);
}
}
return {&fec_packet_mask_[0],
static_cast<size_t>(num_fec_packets * mask_length)};
}
PacketMaskTable::LookUp()主要:
-
在media packet <= 12時，直接從系統預設的兩張表查mask
-
在media packet > 12時，采用間隔的方式進行分組fec
其中查表的過程有一些細節,以kPacketMaskRandomTbl表為例，其定義如下：
const uint8_t kPacketMaskRandomTbl[] = {
12,
1, kMaskRandom1_1,
2, kMaskRandom2_1, kMaskRandom2_2
3, kMaskRandom3_1, kMaskRandom3_2, kMaskRandom3_3
4, kMaskRandom4_1, kMaskRandom4_2, kMaskRandom4_3, kMaskRandom4_4
kPacketMaskRandom5,
kPacketMaskRandom6,
kPacketMaskRandom7,
kPacketMaskRandom8,
kPacketMaskRandom9,
kPacketMaskRandom10,
kPacketMaskRandom11,
kPacketMaskRandom12,
};
以上手動對其中一部分做了宏定義展開，最高12表示該表支持的原始包的長度為12，然后接下來的每一行，是每個原始包的對應數量下不同數量fec包的mask表，比如說kMaskRandom4_3是原始包數量為4fec包數量為3情況下的一個大小為(4x3)掩碼表的起始，理解這個表后，下面的找表函數就好理解了
rtc::ArrayView<const uint8_t> LookUpInFecTable(const uint8_t* table,
int media_packet_index,
int fec_index) {
RTC_DCHECK_LT(media_packet_index, table[0]);
// Skip over the table size.
const uint8_t* entry = &table[1];
uint8_t entry_size_increment = 2; // 0-16 are 2 byte wide, then changes to 6.
// Hop over un-interesting array entries.
// 跳過行
for (int i = 0; i < media_packet_index; ++i) {
if (i == 16)
entry_size_increment = 6;
uint8_t count = entry[0];
++entry; // skip over the count.
for (int j = 0; j < count; ++j) {
entry += entry_size_increment * (j + 1); // skip over the data.
}
}
if (media_packet_index == 16)
entry_size_increment = 6;
RTC_DCHECK_LT(fec_index, entry[0]);
++entry; // Skip over the size.
// Find the appropriate data in the second dimension.
// 跳過列
for (int i = 0; i < fec_index; ++i)
entry += entry_size_increment * (i + 1); // skip over the data.
// 確定目的掩碼表的起始地址和size
size_t size = entry_size_increment * (fec_index + 1);
return {&entry[0], size};
}
2.2.4 webrtc中的Unequal Protection
非均等保護在webrtc中也實現了，但不是使用L1的方式，而采用了另一種方式，在指定的fec包數量下，分配更多的fec包給更重要的包，是通過掩碼表的方式實現Unequal Protection的，這個東西實現在UnequalProtectionMask()
void UnequalProtectionMask(int num_media_packets,
int num_fec_packets,
int num_imp_packets,
int num_mask_bytes,
uint8_t* packet_mask,
PacketMaskTable* mask_table) {
// Set Protection type and allocation
// TODO(marpan): test/update for best mode and some combinations thereof.
ProtectionMode mode = kModeOverlap;
int num_fec_for_imp_packets = 0;
if (mode != kModeBiasFirstPacket) {
num_fec_for_imp_packets = SetProtectionAllocation(
num_media_packets, num_fec_packets, num_imp_packets);
}
int num_fec_remaining = num_fec_packets - num_fec_for_imp_packets;
// Done with setting protection type and allocation
//
// Generate sub_mask1
//
if (num_fec_for_imp_packets > 0) {
ImportantPacketProtection(num_fec_for_imp_packets, num_imp_packets,
num_mask_bytes, packet_mask, mask_table);
}
//
// Generate sub_mask2
//
if (num_fec_remaining > 0) {
RemainingPacketProtection(num_media_packets, num_fec_remaining,
num_fec_for_imp_packets, num_mask_bytes, mode,
packet_mask, mask_table);
}
}
這種unequal就主要體現在給重要保護的包分配多少fec包了，假設現在有m個原始包，n個fec包，m個原始包的前k個是需要重要保護的包，有三種分配方式:
kModeNoOverlap: 分配t個fec包給k個重要原始包做fec，剩下的fec包給剩下的普通原始包 (非重疊版)
kModeOverlap: 分配t個fec包給k個重要原始包做fec，剩下的fec包同時給普通版和重要包做fec (重疊版)
kModeBiasFirstPacket: 全部fec包都對第一個包做編碼。
2.2.5 flexfec
ulpfec在某些丟包的情況下會導致無法恢復, 如下圖所示，p2和p3如果在傳輸的過程中丟失了，就無法通過p1和f1去恢復了
此時如果再引入了列方向上的fec就能夠比較好的解決這個問題,如下圖，對列進行fec后，哪怕p2和p3同時丟失，
也能通過f5和f6對它進行復原, 這就是flexfec, 通過允許更直觀靈活的組包fec應該是flexfec設計的目的，但丟包太多還是存在無法恢復的情況。
2.3 fec調整策略
fec的調整策略在細節上至今都沒有看太明白，只能說一個大概；
fec_rate這個值為fec_packet_num / media_packet_num * 256，設定之后就能確定fec包的個數了，在2.2.3.2中也曾提到過。這個值是綜合碼率，丟包率得到的，具體的計算在VCMFecMethod::ProtectionFactor()中
bool VCMFecMethod::ProtectionFactor(const VCMProtectionParameters* parameters) {
// FEC PROTECTION SETTINGS: varies with packet loss and bitrate
// No protection if (filtered) packetLoss is 0
uint8_t packetLoss = rtc::saturated_cast<uint8_t>(255 * parameters->lossPr);
if (packetLoss == 0) {
_protectionFactorK = 0;
_protectionFactorD = 0;
return true;
}
// Parameters for FEC setting:
// first partition size, thresholds, table pars, spatial resoln fac.
// First partition protection: ~ 20%
uint8_t firstPartitionProt = rtc::saturated_cast<uint8_t>(255 * 0.20);
// Minimum protection level needed to generate one FEC packet for one
// source packet/frame (in RTP sender)
uint8_t minProtLevelFec = 85;
// Threshold on packetLoss and bitRrate/frameRate (=average #packets),
// above which we allocate protection to cover at least first partition.
uint8_t lossThr = 0;
uint8_t packetNumThr = 1;
// Parameters for range of rate index of table.
const uint8_t ratePar1 = 5;
const uint8_t ratePar2 = 49;
// webrtc提供了一個二維表kFecRateTable[rate][loss]，用於查詢不同碼率丟包下的
// fec_rate第一維是碼率，第二維是丟包, 值是fec_rate
// Spatial resolution size, relative to a reference size.
// 計算當前分辨率和參考基分辨率 704 * 576的比
float spatialSizeToRef = rtc::saturated_cast<float>(parameters->codecWidth *
parameters->codecHeight) /
(rtc::saturated_cast<float>(704 * 576));
// resolnFac: This parameter will generally increase/decrease the FEC rate
// (for fixed bitRate and packetLoss) based on system size.
// Use a smaller exponent (< 1) to control/soften system size effect.
// 使用分辨率比得到一個變換系數，用於下文對實際分辨率的變換
// 使用這個變換系數是為了根據
const float resolnFac = 1.0 / powf(spatialSizeToRef, 0.3f);
// 通過parameter計算出每幀的碼率
const int bitRatePerFrame = BitsPerFrame(parameters);
// Average number of packets per frame (source and fec):
const uint8_t avgTotPackets = rtc::saturated_cast<uint8_t>(
1.5f + rtc::saturated_cast<float>(bitRatePerFrame) * 1000.0f /
rtc::saturated_cast<float>(8.0 * _maxPayloadSize));
// FEC rate parameters: for P and I frame
uint8_t codeRateDelta = 0;
uint8_t codeRateKey = 0;
// Get index for table: the FEC protection depends on an effective rate.
// The range on the rate index corresponds to rates (bps)
// from ~200k to ~8000k, for 30fps
// 將碼率進行變換，根據每一幀碼率變換出表的行
const uint16_t effRateFecTable =
rtc::saturated_cast<uint16_t>(resolnFac * bitRatePerFrame);
uint8_t rateIndexTable = rtc::saturated_cast<uint8_t>(
VCM_MAX(VCM_MIN((effRateFecTable - ratePar1) / ratePar1, ratePar2), 0));
// Restrict packet loss range to 50:
// current tables defined only up to 50%
// 只支持50%的丟包
if (packetLoss >= kPacketLossMax) {
packetLoss = kPacketLossMax - 1;
}
// 計算查表得坐標
uint16_t indexTable = rateIndexTable * kPacketLossMax + packetLoss;
// Check on table index
RTC_DCHECK_LT(indexTable, kFecRateTableSize);
// Protection factor for P frame
// 從kFecRateTable[rateIndexTable][packetLoss] 查到對應的factor
codeRateDelta = kFecRateTable[indexTable];
if (packetLoss > lossThr && avgTotPackets > packetNumThr) {
// Set a minimum based on first partition size.
if (codeRateDelta < firstPartitionProt) {
codeRateDelta = firstPartitionProt;
}
}
// Check limit on amount of protection for P frame; 50% is max.
if (codeRateDelta >= kPacketLossMax) {
codeRateDelta = kPacketLossMax - 1;
}
// For Key frame:
// Effectively at a higher rate, so we scale/boost the rate
// The boost factor may depend on several factors: ratio of packet
// number of I to P frames, how much protection placed on P frames, etc.
// 實際碼率上I幀大於P幀，所以會根據I、P幀中的packet數之比進行擴大
const uint8_t packetFrameDelta =
rtc::saturated_cast<uint8_t>(0.5 + parameters->packetsPerFrame);
const uint8_t packetFrameKey =
rtc::saturated_cast<uint8_t>(0.5 + parameters->packetsPerFrameKey);
const uint8_t boostKey = BoostCodeRateKey(packetFrameDelta, packetFrameKey);
rateIndexTable = rtc::saturated_cast<uint8_t>(VCM_MAX(
VCM_MIN(1 + (boostKey * effRateFecTable - ratePar1) / ratePar1, ratePar2),
0));
uint16_t indexTableKey = rateIndexTable * kPacketLossMax + packetLoss;
indexTableKey = VCM_MIN(indexTableKey, kFecRateTableSize);
// Check on table index
assert(indexTableKey < kFecRateTableSize);
// Protection factor for I frame
codeRateKey = kFecRateTable[indexTableKey];
// Boosting for Key frame.
int boostKeyProt = _scaleProtKey * codeRateDelta;
if (boostKeyProt >= kPacketLossMax) {
boostKeyProt = kPacketLossMax - 1;
}
// Make sure I frame protection is at least larger than P frame protection,
// and at least as high as filtered packet loss.
codeRateKey = rtc::saturated_cast<uint8_t>(
VCM_MAX(packetLoss, VCM_MAX(boostKeyProt, codeRateKey)));
// Check limit on amount of protection for I frame: 50% is max.
if (codeRateKey >= kPacketLossMax) {
codeRateKey = kPacketLossMax - 1;
}
// 設置I幀和P幀的fec rate
_protectionFactorK = codeRateKey;
_protectionFactorD = codeRateDelta;
// Generally there is a rate mis-match between the FEC cost estimated
// in mediaOpt and the actual FEC cost sent out in RTP module.
// This is more significant at low rates (small # of source packets), where
// the granularity of the FEC decreases. In this case, non-zero protection
// in mediaOpt may generate 0 FEC packets in RTP sender (since actual #FEC
// is based on rounding off protectionFactor on actual source packet number).
// The correction factor (_corrFecCost) attempts to corrects this, at least
// for cases of low rates (small #packets) and low protection levels.
// 由於fec rate的預估和實際的fec的使用存在出入，這一點在低碼率的時候，fec rate
// 很小時影響很大，在這種情況下，預測一個非0的fec rate，可能導致rtp sender
// 產生-個fec包(因為fec包的實際計算會使用四舍五入)，所以引入了_corrFecCost 0.5
// 去做一個上修正
float numPacketsFl =
1.0f + (rtc::saturated_cast<float>(bitRatePerFrame) * 1000.0 /
rtc::saturated_cast<float>(8.0 * _maxPayloadSize) +
0.5);
const float estNumFecGen =
0.5f +
rtc::saturated_cast<float>(_protectionFactorD * numPacketsFl / 255.0f);
// We reduce cost factor (which will reduce overhead for FEC and
// hybrid method) and not the protectionFactor.
_corrFecCost = 1.0f;
// 對於單packet，預估系數小於產生一個fec包最小系數，預估產生的fec包小於[0.9, 1.1]，引入0.5上修正
if (estNumFecGen < 1.1f && _protectionFactorD < minProtLevelFec) {
_corrFecCost = 0.5f;
}
// 小於0.9 無需上修正
if (estNumFecGen < 0.9f && _protectionFactorD < minProtLevelFec) {
_corrFecCost = 0.0f;
}
// DONE WITH FEC PROTECTION SETTINGS
return true;
}
主要:
- webrtc提供了一個二維表kFecRateTable[rate][loss]，用於查詢不同碼率丟包下的fec_rate, 第一維是碼率，第二維是丟包率, 值是fec_rate
- 計算P幀的碼率，基於參考做一個變換(這個變換暫時理解不了)，根據這個變換后的碼率作為第一碼率，丟包率作為第二維查表得到fec_rate
- 鑒於I幀的碼率比P幀大，使用I幀和P幀的packet數的倍數比，擴大P幀的第一維坐標作為I幀第一維坐標去查表，得到fec_rate
- 由於rtp module實際使用fec_rate對算出來的fec包數會做四舍五入(這點也很奇怪，應該是上取整)，在預估要產生一個fec包，但是預估的fec_rate無法無法達到單packet是最小能產生一個fec包的值，引入一個上修正系數_corrFecCost =0.5(但實際上是沒用上的，因為rtp模塊做的是上取整吧)
3.Ref
1.ULPFEC在WebRTC中的實現 -- weizhenwei
2.webrtc QOS方法二.2（ulpfec rfc5109簡介）--CrystalShaw
5.RFC5109 RTP Payload Format for Generic Forward Error Correction
6. RTP Payload Format for Flexible Forward Error Correction (FlexFEC)