1. 前言
本文是webrtc擁塞控制的下文，主要介紹的是從cc-controller獲取碼率之后，如何將碼率設置到PacingController控制發送速率，同時如何將碼率分配調整到各個stream，各個stream的layer, simulcast，fec中
2. 正文
2.1 整體碼控結構
webrtc中是會同時存在多個stream，但所有的stream都會共用一個碼率預估和平滑發送，這很符合邏輯(雖然gcc保障帶寬公平性)，發送數據的流程如上圖數字所標，不同的stream最終發包的時候都是發送到RtpControllerSend的PacketSender中，然后在PacingController進行碼率控制發送，發送包經過PacketRouter路由到其所屬的RTP_RTCP進行緩存記錄以便重傳，然后通過RtpSenderEgress將包發送到Transport層，由Transport做Dtls，通過ice發到網絡。
在擁塞控制(上)中已經介紹完了整個CcController通過cc-feedback計算目標碼率，由cc-controller計算出來的碼率最后會被放到PacingController中控制發送速度，同時計算出來的碼率也會由RtpControllerSend通知觀察者回調至上層做碼率調整
整個調節過程如上圖所示，會將cc-controller估算出來的碼率分配給所有的streams，每個stream有設置好的最小碼率(min_bitrate)和最大碼率(max_bitrate)通過它們進行分配，本文沒有介紹音頻部分的分配和處理，只介紹了視頻部分的，對於每個VideoStream，碼率有兩個部分構成，一個是media(媒體)，一個是Protection(抗丟包)
media(媒體碼率)由Overhead(Rtp頭，UDP頭開銷), Packetization(編碼媒體幀封裝開銷)，Encoder(編碼碼率)構成；
而Encoder(編碼碼率)由可以根據simulcast進行細分，每個simulcast下還有不同的Temporal
Protection(抗丟包碼率)主要由Fec碼率和Rtx(nack重傳碼率)構成
下面逐漸對以上內容進行細述
2.2 RtpTransportController 反饋碼率結果
在擁塞控制(上)中已經介紹完了整個CcController通過cc-feedback計算目標碼率，RtpTransportControllerSend從CcController獲取目標碼率后會使用PostUpdates()進行更新
void RtpTransportControllerSend::OnTransportFeedback(
const rtcp::TransportFeedback& feedback) {
feedback_demuxer_.OnTransportFeedback(feedback);
auto feedback_time = Timestamp::Millis(clock_->TimeInMilliseconds());
task_queue_.PostTask([this, feedback, feedback_time]() {
RTC_DCHECK_RUN_ON(&task_queue_);
absl::optional<TransportPacketsFeedback> feedback_msg =
transport_feedback_adapter_.ProcessTransportFeedback(feedback,
feedback_time);
if (feedback_msg && controller_) {
//從cc-controller中獲取目標碼率進行設置
PostUpdates(controller_->OnTransportPacketsFeedback(*feedback_msg));
}
pacer()->UpdateOutstandingData(
transport_feedback_adapter_.GetOutstandingData());
});
}
PostUpdates()的過程如下所示:
void RtpTransportControllerSend::PostUpdates(NetworkControlUpdate update) {
if (update.congestion_window) {
// 設置擁塞窗口大小
pacer()->SetCongestionWindow(*update.congestion_window);
}
if (update.pacer_config) {
// 設置平滑發送速率
pacer()->SetPacingRates(update.pacer_config->data_rate(),
update.pacer_config->pad_rate());
}
for (const auto& probe : update.probe_cluster_configs) {
// 按照probe_cluster_config生成探測簇
pacer()->CreateProbeCluster(probe.target_data_rate, probe.id);
}
if (update.target_rate) {
// 目標碼率更新了，生成目標碼率
control_handler_->SetTargetRate(*update.target_rate);
// 更新碼率分配
UpdateControlState();
}
}
RtpTransportControllerSend::PostUpdates()中:
-
調用SetCongestionWindow()設置pacer的擁塞窗口大小, 詳見2.3.1
-
調用SetPacingRates()設置pacer的發送速率，詳見2.3.2
-
調用CreateProbeCluster()設置pacer的探測簇, 詳見2.3.3
-
調用UpdateControlState()將碼率反饋給上層源端，去做編碼，simulcast, svc，fec的碼率控制。
2.3 PacingController 控制發送碼率
2.3.1 擁塞窗口
PacingController使用了一個擁塞窗口來輔助平滑發送，這個擁塞窗口的值在擁塞控制(上)-2.8中提到過，其內涵是一個使用目標碼率 * rtt計算的值。
PacingController使用該擁塞窗口來控制一個rtt內的數據發送，當檢測到當前的發送中的數據(已發送但是未得到ack的數據)大於該窗口值得時候，就處於一個congested狀態。
bool PacingController::Congested() const {
if (congestion_window_size_.IsFinite()) {
// 發送中的數據(outstanding_data_)大於當前窗口值，當前處於擁塞狀態
return outstanding_data_ >= congestion_window_size_;
}
return false;
}
2.3.1.1 outstanding_data的維護
發送中的數據(outstanding_data)大小的更新過程在兩個地方:
outstanding_data第一個更新在底層udp socket成功發送數據的時候會執行回調OnPacketSent()通知發送了多少數據
/**
* @description: packet成功從udp發送出去的回調，包此時未到達接收端
* @param {packet_size} 發送的數據大小
* @param {send_time} 發送的時間
* @return {void}
*/
void PacingController::OnPacketSent(RtpPacketMediaType packet_type,
DataSize packet_size,
Timestamp send_time) {
if (!first_sent_packet_time_) {
first_sent_packet_time_ = send_time;
}
bool audio_packet = packet_type == RtpPacketMediaType::kAudio;
if (!audio_packet || account_for_audio_) {
// 更新已發送的數據大小
UpdateBudgetWithSentData(packet_size);
}
last_send_time_ = send_time;
last_process_time_ = send_time;
}
其中調用的UpdateBudgetWithSentData()如下：
void PacingController::UpdateBudgetWithSentData(DataSize size) {
// 增加發送中的數據量
outstanding_data_ += size;
if (mode_ == ProcessMode::kPeriodic) {
// 數據已發送,降低發送budget
media_budget_.UseBudget(size.bytes());
padding_budget_.UseBudget(size.bytes());
} else {
media_debt_ += size;
media_debt_ = std::min(media_debt_, media_rate_ * kMaxDebtInTime);
padding_debt_ += size;
padding_debt_ = std::min(padding_debt_, padding_rate_ * kMaxDebtInTime);
}
}
其中:
- 主要關注最上面的outstang_data的更新
- 其次，pacing controller使用了周期(KPeriodic)和動態(KDynamic)兩種方式管理媒體發送速度，后文再詳述
outstanding_data第二個更新主要是通過PacingController::UpdateOutstandingData()直接做數值上的更新的
void PacingController::UpdateOutstandingData(DataSize outstanding_data) {
const bool was_congested = Congested();
outstanding_data_ = outstanding_data;//直接更新
if (was_congested && !Congested()) {
TimeDelta elapsed_time = UpdateTimeAndGetElapsed(CurrentTime());
UpdateBudgetWithElapsedTime(elapsed_time);
}
}
這個調用會在socket層探測到當前網絡的發生變化的時候，會直接對outstand_data重置為0
void RtpTransportControllerSend::OnNetworkRouteChanged(
....
task_queue_.PostTask([this, msg, network_route] {
....
// 網絡路由發生了改變，將發送中的數據重設為0
pacer()->UpdateOutstandingData(DataSize::Zero());
});
}
}
void RtpTransportControllerSend::OnNetworkAvailability(bool network_available) {
....
task_queue_.PostTask([this, msg]() {
....
// 網絡的可用性發生改變，對發送中的數據大小進行重新初始化為0
pacer()->UpdateOutstandingData(DataSize::Zero());
....
});
....
}
或者在收到了cc-feedback或者RR-report的時候，在這里它們它們充當了一個ack的作用，通過它們就可以將發送中的數據更新降低, 舉cc-feedback為例：
void RtpTransportControllerSend::OnTransportFeedback(
const rtcp::TransportFeedback& feedback) {
....
task_queue_.PostTask([this, feedback, feedback_time]() {
// 解析cc-feedback獲得feedback_msg, 更新transport_feedback_adapter_
absl::optional<TransportPacketsFeedback> feedback_msg =
transport_feedback_adapter_.ProcessTransportFeedback(feedback,
feedback_time);
// 更新發送中的數據大小
pacer()->UpdateOutstandingData(
transport_feedback_adapter_.GetOutstandingData());
});
}
其中transport_feedback_adapter使用變量in_filght對發送數據進行統計維護
DataSize TransportFeedbackAdapter::GetOutstandingData() const {
return in_flight_.GetOutstandingData(network_route_);
}
in_flight詳細維護過程如下，發送數據的時候會在TransportFeedbackAdapter::ProcessSentPacket()增加in_filght的大小
absl::optional<SentPacket> TransportFeedbackAdapter::ProcessSentPacket(
const rtc::SentPacket& sent_packet) {
....
if (sent_packet.info.included_in_feedback || sent_packet.packet_id != -1) {
....
if (!packet_retransmit) {
if (it->second.sent.sequence_number > last_ack_seq_num_)
in_flight_.AddInFlightPacketBytes(it->second);// 新增已發送數據
}
}
}
....
}
收到feedback的時候會調用TransportFeedbackAdapter::ProcessTransportFeedbackInner()，對當前已經ack到的seq從in_flight中刪除
TransportFeedbackAdapter::ProcessTransportFeedbackInner(
const rtcp::TransportFeedback& feedback,
Timestamp feedback_receive_time) {
for (const auto& packet : feedback.GetAllPackets()) {
int64_t seq_num = seq_num_unwrapper_.Unwrap(packet.sequence_number());
if (seq_num > last_ack_seq_num_) {
// Starts at history_.begin() if last_ack_seq_num_ < 0, since any valid
// sequence number is >= 0.
for (auto it = history_.upper_bound(last_ack_seq_num_);
it != history_.upper_bound(seq_num); ++it) {
// 將上一個包到已經ack的當前包的一個in_flight刪除
in_flight_.RemoveInFlightPacketBytes(it->second);
}
last_ack_seq_num_ = seq_num;
}
}
2.3.1.2 擁塞狀態的使用
從congested()獲取到的擁塞狀態主要在PacingController控制下次發送時間的時候起作用，NextSendTime()是PacingController給PacedSender的建議下次發包時間，還未到這個時間PacedSender陷入睡眠，當到達這個時間后會讓PacingController起來工作，處理隊列中的包。
其中就看到了當判斷到當前處於一個擁塞狀態的時候，就會返回擁塞沉睡間隔，暫時不處理隊列中的包。
Timestamp PacingController::NextSendTime() const {
....
if (Congested() || packet_counter_ == 0) {
// We need to at least send keep-alive packets with some interval.
return last_send_time_ + kCongestedPacketInterval;
}
....
}
此外在發送的時候，也會判斷擁塞窗口是否已經使用完畢，如果是，則不發包，詳見2.3.2.2
2.3.2 PacingRate
2.3.2.1 PacingRate的更新
Pacing相關碼率的更新會通過PacingController::SetPacingRates()進行
void PacingController::SetPacingRates(DataRate pacing_rate,
DataRate padding_rate) {
RTC_DCHECK_GT(pacing_rate, DataRate::Zero());
media_rate_ = pacing_rate;
padding_rate_ = padding_rate;
pacing_bitrate_ = pacing_rate;
padding_budget_.set_target_rate_kbps(padding_rate.kbps());
RTC_LOG(LS_VERBOSE) << "bwe:pacer_updated pacing_kbps="
<< pacing_bitrate_.kbps()
<< " padding_budget_kbps=" << padding_rate.kbps();
}
PacingController::SetPacingRates()中主要設置的是pacing_rate和padding_rate，這兩個值在GoogCcNetworkController::GetPacingRates()設置的，前者是cc-controller最終給出的目標碼率 * 系數, 而后者是一個設置值，這個值還沒細看不知道是否動態更新，但只有OnStreamConfig()的時候才設置了一次，感覺是應該不是動態更新
PacerConfig GoogCcNetworkController::GetPacingRates(Timestamp at_time) const {
// Pacing rate is based on target rate before congestion window pushback,
// because we don't want to build queues in the pacer when pushback occurs.
// 此處的pacing rate使用的是last_loss_based_target_rate_, 這個值沒有經過擁塞窗口的更新處理
// 但是沒太看懂注釋，"當退避產生的時候不想在pacer創建隊列"，因為pacer有兩種,一種是有queue的
// 一種是無queue的，可能想要表達的是congestion push back不應用在有queue的隊列上?
DataRate pacing_rate =
std::max(min_total_allocated_bitrate_, last_loss_based_target_rate_) *
pacing_factor_;
// padding_rate 主要的值還是max_padding_rate_，這是一個來自於外部(bitrateAllocation)計算的一個值
// 其次，它肯定不能大於窗口控制的碼率(last_pushback_target_rate_)
DataRate padding_rate =
std::min(max_padding_rate_, last_pushback_target_rate_);
PacerConfig msg;
msg.at_time = at_time;
msg.time_window = TimeDelta::Seconds(1);//1s
msg.data_window = pacing_rate * msg.time_window;
msg.pad_window = padding_rate * msg.time_window;
return msg;
}
2.3.2.2 PacingRate的使用
PacingRate是用來控制發送速度的，但PacingController並沒有直接使用它，PacingController中基於PacingRate為了控制方便，封裝了兩個類用來控制發送速度，一個是(class IntervalBudget)，一個是media_debt(DataSize)，前者使用在PacingController的Periodic模式中，在該模式下，PacingController會周期間隔(5ms)的去處理包隊列，后者使用在是Dynamic模式，在該模式下，PacingController處理包隊列的間隔是任意的，media_budget和media_debt的值會隨着時間的流逝而變化；下面以media_budget為例介紹
PacingRate在PacingController::ProcessPackets()中被設置到media_budget上, 並在下面的while循環中使用GetPendingPacket()從隊列獲取包去發送的時候會使用到:
void PacingController::ProcessPackets() {
....
if (elapsed_time > TimeDelta::Zero()) {
// 獲取pacing rate
DataRate target_rate = pacing_bitrate_;
....
if (mode_ == ProcessMode::kPeriodic) {
// 將pacing rate設置到media_budget中
media_budget_.set_target_rate_kbps(target_rate.kbps());
// 基於流逝的時間更新budget
UpdateBudgetWithElapsedTime(elapsed_time);
} else {
media_rate_ = target_rate;
}
}
....
while (!paused_) {
....
// 根據budget和發送時間從queue中獲取下一個要發的包
std::unique_ptr<RtpPacketToSend> rtp_packet =
GetPendingPacket(pacing_info, target_send_time, now);
....
}
....
}
PacingController::ProcessPackets() 中
-
會調用
set_target_rate_kbps()將pacing rate更新到media_budget的目標碼率中 -
基於流逝的時間使用
UpdateBudgetWithElapsedTime()去增大media_budget 的值void PacingController::UpdateBudgetWithElapsedTime(TimeDelta delta) { if (mode_ == ProcessMode::kPeriodic) { delta = std::min(kMaxProcessingInterval, delta); media_budget_.IncreaseBudget(delta.ms()); // old + target_bitrate * delate_ms padding_budget_.IncreaseBudget(delta.ms()); } else { media_debt_ -= std::min(media_debt_, media_rate_ * delta); padding_debt_ -= std::min(padding_debt_, padding_rate_ * delta); } } -
然后在發包的時候調用PacingController::GetPendingPacket()隊列中待發送包的時候會通過media_budget判斷當前是否還有足夠的發送預算
std::unique_ptr<RtpPacketToSend> PacingController::GetPendingPacket( const PacedPacketInfo& pacing_info, Timestamp target_send_time, Timestamp now) { if (packet_queue_.Empty()) { return nullptr; } bool unpaced_audio_packet = !pace_audio_ && packet_queue_.LeadingAudioPacketEnqueueTime().has_value(); bool is_probe = pacing_info.probe_cluster_id != PacedPacketInfo::kNotAProbe; if (!unpaced_audio_packet && !is_probe) { if (Congested()) { // 擁塞窗口滿了，不發包 return nullptr; } // 沒有budget了，不發包 if (mode_ == ProcessMode::kPeriodic) { if (media_budget_.bytes_remaining() <= 0) { // Not enough budget. return nullptr; } } else { // Dynamic processing mode. if (now <= target_send_time) { // target_send_time是基於當前隊列大小/目標發送碼率(media_rate)得到的一個目標發送 // 時間值 // We allow sending slightly early if we think that we would actually // had been able to, had we been right on time - i.e. the current debt // is not more than would be reduced to zero at the target sent time. TimeDelta flush_time = media_debt_ / media_rate_; if (now + flush_time > target_send_time) { return nullptr; } } } } return packet_queue_.Pop(); }
而media_budget的減少在之前的PacingController::UpdateBudgetWithSentData()中已經提到過，在 packet成功發送的時候會調用onPacketSent，則會被消耗，最終調用UpdateBudgetWithSentData()去減少
void PacingController::UpdateBudgetWithSentData(DataSize size) {
outstanding_data_ += size;
if (mode_ == ProcessMode::kPeriodic) {
// 數據已發送,降低發送budget
media_budget_.UseBudget(size.bytes());
padding_budget_.UseBudget(size.bytes());
} else {
media_debt_ += size;
media_debt_ = std::min(media_debt_, media_rate_ * kMaxDebtInTime);
padding_debt_ += size;
padding_debt_ = std::min(padding_debt_, padding_rate_ * kMaxDebtInTime);
}
}
2.3.3 ProbeCluster
PacingController除了控制發送速率，同時還肩負另一個功能，支持碼率探測，其使用了一個叫做BitratePorber 的類以用來響應cc-controller的探測需求，BitratePorber在擁塞控制(上)-碼率預估的2.4.2已經提到過了，此處不再贅述。
PacingController部分的內容到此結束了，這是cc-controller估算出來的碼率對下游最終發送上的應用，接下來介紹估算出來的碼率對上游編碼和fec等的影響
2.4 將碼率分配到源
void RtpTransportControllerSend::PostUpdates(NetworkControlUpdate update) {
....
if (update.target_rate) {
// 通知上層
control_handler_->SetTargetRate(*update.target_rate);
UpdateControlState();
}
}
在RtpTransportControllerSend::PostUpdates()的最后：
-
將目標碼率設置到了control_handler_
-
調用UpdateControlState()，通過control_handler獲取目標碼率，並將最新碼率回調給觀察者上層
void RtpTransportControllerSend::UpdateControlState() { absl::optional<TargetTransferRate> update = control_handler_->GetUpdate(); if (!update) return; retransmission_rate_limiter_.SetMaxRate(update->target_rate.bps()); RTC_DCHECK(observer_ != nullptr); observer_->OnTargetTransferRate(*update); }
上層觀察者(call)接收碼率變換通知的是Call::OnTargetTransferRate()
void Call::OnTargetTransferRate(TargetTransferRate msg) {
RTC_DCHECK_RUN_ON(send_transport_queue());
uint32_t target_bitrate_bps = msg.target_rate.bps();
// For controlling the rate of feedback messages.
// 控制feedback的開銷在5%左右
receive_side_cc_.OnBitrateChanged(target_bitrate_bps);
// 重新分配到各個stream中的碼流
bitrate_allocator_->OnNetworkEstimateChanged(msg);
// 更新統計直方圖
worker_thread_->PostTask(
ToQueuedTask(task_safety_, [this, target_bitrate_bps]() {
RTC_DCHECK_RUN_ON(worker_thread_);
last_bandwidth_bps_ = target_bitrate_bps;
// Ignore updates if bitrate is zero (the aggregate network state is
// down) or if we're not sending video.
if (target_bitrate_bps == 0 || video_send_streams_.empty()) {
estimated_send_bitrate_kbps_counter_.ProcessAndPause();
pacer_bitrate_kbps_counter_.ProcessAndPause();
return;
}
estimated_send_bitrate_kbps_counter_.Add(target_bitrate_bps / 1000);
// Pacer bitrate may be higher than bitrate estimate if enforcing min
// bitrate.
uint32_t pacer_bitrate_bps =
std::max(target_bitrate_bps, min_allocated_send_bitrate_bps_);
pacer_bitrate_kbps_counter_.Add(pacer_bitrate_bps / 1000);
}));
}
Call::OnTargetTransferRate()中:
-
調用 ReceiveSideCongestionController::OnBitrateChanged(), 用於更新cc-report部分的發送時間間隔，將這一部分的發送碼率控制在目標碼率的5%。
void RemoteEstimatorProxy::OnBitrateChanged(int bitrate_bps) { // TwccReportSize = Ipv4(20B) + UDP(8B) + SRTP(10B) + // AverageTwccReport(30B) // TwccReport size at 50ms interval is 24 byte. // TwccReport size at 250ms interval is 36 byte. // AverageTwccReport = (TwccReport(50ms) + TwccReport(250ms)) / 2 constexpr int kTwccReportSize = 20 + 8 + 10 + 30; const double kMinTwccRate = kTwccReportSize * 8.0 * 1000.0 / send_config_.max_interval->ms(); const double kMaxTwccRate = kTwccReportSize * 8.0 * 1000.0 / send_config_.min_interval->ms(); // Let TWCC reports occupy 5% of total bandwidth. // bandwidth_fraction 為 0.05 MutexLock lock(&lock_); send_interval_ms_ = static_cast<int>( 0.5 + kTwccReportSize * 8.0 * 1000.0 / rtc::SafeClamp(send_config_.bandwidth_fraction * bitrate_bps, kMinTwccRate, kMaxTwccRate)); } -
調用BitrateAllocator::OnNetworkEstimateChanged()重新將碼率分配到各個流中, BitrateAllocator中的內容比較多，會在2.5展開介紹
-
最后還Post一個任務，這個任務會將發送碼率更新到統計直方圖
2.5 BitrateAllocator 分配碼率
BitrateAllocator是對碼率進行重新分配的分配器，邏輯入口為BitrateAllocator::OnNetworkEstimateChanged()
其會把碼率分配到不同的stream上
void BitrateAllocator::OnNetworkEstimateChanged(TargetTransferRate msg) {
RTC_DCHECK_RUN_ON(&sequenced_checker_);
// 獲取目標碼率和保守碼率
last_target_bps_ = msg.target_rate.bps();
last_stable_target_bps_ = msg.stable_target_rate.bps();
last_non_zero_bitrate_bps_ =
last_target_bps_ > 0 ? last_target_bps_ : last_non_zero_bitrate_bps_;
// loss_ration 放大255
int loss_ratio_255 = msg.network_estimate.loss_rate_ratio * 255;
last_fraction_loss_ =
rtc::dchecked_cast<uint8_t>(rtc::SafeClamp(loss_ratio_255, 0, 255));
last_rtt_ = msg.network_estimate.round_trip_time.ms();
last_bwe_period_ms_ = msg.network_estimate.bwe_period.ms();
// Periodically log the incoming BWE.
int64_t now = msg.at_time.ms();
if (now > last_bwe_log_time_ + kBweLogIntervalMs) {
RTC_LOG(LS_INFO) << "Current BWE " << last_target_bps_;
last_bwe_log_time_ = now;
}
// 按照目標碼率(target_bitrate)為所有的stream分配碼率
auto allocation = AllocateBitrates(allocatable_tracks_, last_target_bps_);
// 按照保守目標碼率(stable_target)為所有的stream分配碼率
auto stable_bitrate_allocation =
AllocateBitrates(allocatable_tracks_, last_stable_target_bps_);
for (auto& config : allocatable_tracks_) {
uint32_t allocated_bitrate = allocation[config.observer];
uint32_t allocated_stable_target_rate =
stable_bitrate_allocation[config.observer];
BitrateAllocationUpdate update;
update.target_bitrate = DataRate::BitsPerSec(allocated_bitrate);
update.stable_target_bitrate =
DataRate::BitsPerSec(allocated_stable_target_rate);
update.packet_loss_ratio = last_fraction_loss_ / 256.0;
update.round_trip_time = TimeDelta::Millis(last_rtt_);
update.bwe_period = TimeDelta::Millis(last_bwe_period_ms_);
update.cwnd_reduce_ratio = msg.cwnd_reduce_ratio;
// 更新流上的碼率
uint32_t protection_bitrate = config.observer->OnBitrateUpdated(update);
if (allocated_bitrate == 0 && config.allocated_bitrate_bps > 0) {
if (last_target_bps_ > 0)
++num_pause_events_;
// The protection bitrate is an estimate based on the ratio between media
// and protection used before this observer was muted.
uint32_t predicted_protection_bps =
(1.0 - config.media_ratio) * config.config.min_bitrate_bps;
RTC_LOG(LS_INFO) << "Pausing observer " << config.observer
<< " with configured min bitrate "
<< config.config.min_bitrate_bps
<< " and current estimate of " << last_target_bps_
<< " and protection bitrate "
<< predicted_protection_bps;
} else if (allocated_bitrate > 0 && config.allocated_bitrate_bps == 0) {
if (last_target_bps_ > 0)
++num_pause_events_;
RTC_LOG(LS_INFO) << "Resuming observer " << config.observer
<< ", configured min bitrate "
<< config.config.min_bitrate_bps
<< ", current allocation " << allocated_bitrate
<< " and protection bitrate " << protection_bitrate;
}
// Only update the media ratio if the observer got an allocation.
if (allocated_bitrate > 0)
config.media_ratio = MediaRatio(allocated_bitrate, protection_bitrate);
config.allocated_bitrate_bps = allocated_bitrate;
}
// 更新limit,編碼對一些特地分辨率的設置的預設值，當編碼器發生調整的
// 時候，limit是會發生改變的
UpdateAllocationLimits();
}
BitrateAllocator::OnNetworkEstimateChanged()中:
- 首先從TargetTransferRate獲取目標碼率和保守碼率，然后調用AllocateBitrates()將碼率分配到各個stream中， AllocateBitrates()在2.5.1中會詳細介紹
- 然后調用observer->OnBitrateUpdated()將分配的結果發送到各個流中，音頻的處理函數為AudioSendStream::OnBitrateUpdated()， 視頻的處理函數為VideoSendStreamImpl::OnBitrateUpdated()，這一塊在2.5.2中展開詳細介紹
- 給流分配的碼率的也會導致幀率和碼率的調整，從而影響到每個流的min_bitrate和max_bitrate(對於編碼的min_bitrate和max_bitrate，webrtc中兩種方式設置，一種是從trial中獲得，一種是對於一些分辨率的mo人設置，詳細的可參考： encoder_info_setting.cc)，最終會調用UpdateAllocationLimits()計算總的能分配到的min_bitrate和max_bitrate，通過GoogCcNetworkController::OnStreamsConfig()傳到cc-controller中去更新SendSideBandwidthEstimation，讓其再次做碼率預估，看之前預估的碼率是否需要修改，重新通知整個系統進行更改，這一塊將會在2.5.3中詳細介紹
2.5.1 AllocateBitrates()
AllocateBitrates()會將碼率分配到所有的stream(audio stream, video stream)，主要是根據每個流config中的min_bitrate_bps和max_bitrate_bps進行分配。
std::map<BitrateAllocatorObserver*, int> AllocateBitrates(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate) {
if (allocatable_tracks.empty())
return std::map<BitrateAllocatorObserver*, int>();
if (bitrate == 0)
return ZeroRateAllocation(allocatable_tracks);
// 統計所有track設定的最大最小碼率
uint32_t sum_min_bitrates = 0;
uint32_t sum_max_bitrates = 0;
for (const auto& observer_config : allocatable_tracks) {
sum_min_bitrates += observer_config.config.min_bitrate_bps;
sum_max_bitrates += observer_config.config.max_bitrate_bps;
}
// Not enough for all observers to get an allocation, allocate according to:
// enforced min bitrate -> allocated bitrate previous round -> restart paused
// streams.
// 碼率不夠，按照最低分配的策略，給每個流分配碼率
if (!EnoughBitrateForAllObservers(allocatable_tracks, bitrate,
sum_min_bitrates))
return LowRateAllocation(allocatable_tracks, bitrate);
// All observers will get their min bitrate plus a share of the rest. This
// share is allocated to each observer based on its bitrate_priority.
if (bitrate <= sum_max_bitrates)
return NormalRateAllocation(allocatable_tracks, bitrate, sum_min_bitrates);
// All observers will get up to transmission_max_bitrate_multiplier_ x max.
return MaxRateAllocation(allocatable_tracks, bitrate, sum_max_bitrates);
}
在AllocateBitrates()中:
- 首先遍歷所有的track，統計出sum_min_bitrates和sum_max_bitrates
- 調用EnoughBitrateForAllObservers()判斷目標碼率是否能夠滿足分配到所有的流，如果不是則調用LowRateAllocation()進行最小碼率分配，否則的話檢查當前的碼率是否小於最大所需碼率，是的話調用NormalRateAllocation()進行分配，當超過了最大所需碼率后，則調用MaxRateAllocation()進行分配。
EnoughBitrateForAllObservers()如下：
bool EnoughBitrateForAllObservers(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
uint32_t sum_min_bitrates) {
if (bitrate < sum_min_bitrates)
return false;
// 計算出多余的碼流可以平均攤給每個track的為多少
uint32_t extra_bitrate_per_observer =
(bitrate - sum_min_bitrates) /
static_cast<uint32_t>(allocatable_tracks.size());
// 算上多余均攤之后，無法滿足每個track所需的碼率(考慮上了Hysteresis即暫停滯后過所需要的增益碼率),則false
for (const auto& observer_config : allocatable_tracks) {
if (observer_config.config.min_bitrate_bps + extra_bitrate_per_observer <
observer_config.MinBitrateWithHysteresis()) {
return false;
}
}
return true;
}
EnoughBitrateForAllObservers()中:
-
先按照每個流的最低碼率進行分配，然后將多余的碼率平均分配到每個流上，但如果在這種分配模式下流的碼率仍然無法達到最低的滯后碼率，則認為當前的碼率不足；流的滯后(Hysteresis)是該率在上一次分配中由於碼率有限被分配到了0碼率而導致暫停，此刻它的最低滯后碼率將會在最低碼率的基礎上增加10%
/** * @description: 獲取最小的碼率(會考慮過流是否暫停過，如果是，增益10%) * @param {*} * @return {*} */ uint32_t bitrate_allocator_impl::AllocatableTrack::MinBitrateWithHysteresis() const { uint32_t min_bitrate = config.min_bitrate_bps; if (LastAllocatedBitrate() == 0) { // 被暫停發送了要追加一個10%的系數提升 min_bitrate += std::max(static_cast<uint32_t>(kToggleFactor * min_bitrate), kMinToggleBitrateBps); } // Account for protection bitrate used by this observer in the previous // allocation. // Note: the ratio will only be updated when the stream is active, meaning a // paused stream won't get any ratio updates. This might lead to waiting a bit // longer than necessary if the network condition improves, but this is to // avoid too much toggling. // 啟用了fec，需要通過media_ratio 把fec的碼率也計算進去，其中media_ratio就是媒體所占分配碼率部分 // media_ratio = allocated_bitrate - protection_bitrate / allocated_bitrate if (media_ratio > 0.0 && media_ratio < 1.0) min_bitrate += min_bitrate * (1.0 - media_ratio); return min_bitrate; }
碼率不足的時候會調用LowRateAllocation()進行碼率分配
std::map<BitrateAllocatorObserver*, int> LowRateAllocation(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate) {
std::map<BitrateAllocatorObserver*, int> allocation;
// Start by allocating bitrate to observers enforcing a min bitrate, hence
// remaining_bitrate might turn negative.
// 先每個流分配一個強制最小碼率(enforce_min_bitrate)
int64_t remaining_bitrate = bitrate;
for (const auto& observer_config : allocatable_tracks) {
int32_t allocated_bitrate = 0;
if (observer_config.config.enforce_min_bitrate)
allocated_bitrate = observer_config.config.min_bitrate_bps;
allocation[observer_config.observer] = allocated_bitrate;
remaining_bitrate -= allocated_bitrate;
}
// Allocate bitrate to all previously active streams.
// 如果還有碼率，則為活躍(不暫停)的流分配碼率
if (remaining_bitrate > 0) {
for (const auto& observer_config : allocatable_tracks) {
// 配置了enforce_min_bitrate或者當前處於暫停的流，先跳過
if (observer_config.config.enforce_min_bitrate ||
observer_config.LastAllocatedBitrate() == 0)
continue;
// 獲取需要的碼率(包含fec)
uint32_t required_bitrate = observer_config.MinBitrateWithHysteresis();
if (remaining_bitrate >= required_bitrate) {
allocation[observer_config.observer] = required_bitrate;
remaining_bitrate -= required_bitrate;
}
}
}
// Allocate bitrate to previously paused streams.
// 如果還有碼率，則為暫停的流分配碼率
if (remaining_bitrate > 0) {
for (const auto& observer_config : allocatable_tracks) {
if (observer_config.LastAllocatedBitrate() != 0)
continue;
// Add a hysteresis to avoid toggling.
uint32_t required_bitrate = observer_config.MinBitrateWithHysteresis();
if (remaining_bitrate >= required_bitrate) {
allocation[observer_config.observer] = required_bitrate;
remaining_bitrate -= required_bitrate;
}
}
}
// Split a possible remainder evenly on all streams with an allocation.
// 如果還有剩余的碼率，則均分給所有的流
if (remaining_bitrate > 0)
DistributeBitrateEvenly(allocatable_tracks, remaining_bitrate, false, 1,
&allocation);
RTC_DCHECK_EQ(allocation.size(), allocatable_tracks.size());
return allocation;
}
LowRateAllocation()中：
- 先每個流分配一個強制最小碼率(enforce_min_bitrate)
- 如果還有碼率，則為活躍(LastAllocatedBitrate不為0)的流分配碼率
- 如果還有碼率，則為暫停的流分配碼率
- 如果還有碼率，再均分給所有的流
關於這個策略的第二點要非常注意，這種分配模式下將可能導致碼率低的時候大部分流勉強正常工作，而某部分流持續異常，出現這種情況，可以檢查一下帶寬是否太低了。
碼率充足的時候會調用NormalRateAllocation()進行分配
std::map<BitrateAllocatorObserver*, int> NormalRateAllocation(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
uint32_t sum_min_bitrates) {
// 為每個流分配最小滿足碼率，並計算剩余容量
std::map<BitrateAllocatorObserver*, int> allocation;
std::map<BitrateAllocatorObserver*, int> observers_capacities;
for (const auto& observer_config : allocatable_tracks) {
// 分配滿足最小碼率
allocation[observer_config.observer] =
observer_config.config.min_bitrate_bps;
// 每個stream的剩余能被分配的容量
observers_capacities[observer_config.observer] =
observer_config.config.max_bitrate_bps -
observer_config.config.min_bitrate_bps;
}
bitrate -= sum_min_bitrates;
// TODO(srte): Implement fair sharing between prioritized streams, currently
// they are treated on a first come first serve basis.
// 所有的流都會有一個優先分配碼率，此處是先將碼率分配給有優先分配碼率的流
for (const auto& observer_config : allocatable_tracks) {
// 計算剩余的碼率
int64_t priority_margin = observer_config.config.priority_bitrate_bps -
allocation[observer_config.observer];
if (priority_margin > 0 && bitrate > 0) {
int64_t extra_bitrate = std::min<int64_t>(priority_margin, bitrate);
allocation[observer_config.observer] +=
rtc::dchecked_cast<int>(extra_bitrate);
observers_capacities[observer_config.observer] -= extra_bitrate;
bitrate -= extra_bitrate;
}
}
// From the remaining bitrate, allocate a proportional amount to each observer
// above the min bitrate already allocated.
// 基於各個流的優先碼率比例對對剩余的碼率進行分配
if (bitrate > 0)
DistributeBitrateRelatively(allocatable_tracks, bitrate,
observers_capacities, &allocation);
return allocation;
}
NormalRateAllocation()中:
- 首先會滿足所有track的最小碼率(min_bitrate_bps)，然后通過max_bitrate_bps - min_bitrate_bps計算了一個值capacities，表示該流還能夠被分配的容量
- 如果還有碼率能被分配，遍歷所有的流，將分配給流的碼率提升到priority_bitrate，這個值目前僅在音頻中見到會通過field trial的方式設置，視頻中為0，我把它稱之為流的優先碼率，相對於沒有設置該值的流，設置了的能夠優先獲得更高的碼率
- 如果還有碼率剩余，則調用DistributeBitrateRelatively()，將剩余的碼率基於各個流的優先碼率的比進行分配
當能被分配的碼率超過最大的時候則直接調用MaxRateAllocation()按照最大的分配
std::map<BitrateAllocatorObserver*, int> MaxRateAllocation(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
uint32_t sum_max_bitrates) {
std::map<BitrateAllocatorObserver*, int> allocation;
// 按照max_bitrate_bps進行分配
for (const auto& observer_config : allocatable_tracks) {
allocation[observer_config.observer] =
observer_config.config.max_bitrate_bps;
bitrate -= observer_config.config.max_bitrate_bps;
}
// 將剩余得碼率均分給所有流，但不超過流最大碼率得2倍
DistributeBitrateEvenly(allocatable_tracks, bitrate, true,
kTransmissionMaxBitrateMultiplier, &allocation);
return allocation;
}
void DistributeBitrateEvenly(
const std::vector<AllocatableTrack>& allocatable_tracks,
uint32_t bitrate,
bool include_zero_allocations,
int max_multiplier,
std::map<BitrateAllocatorObserver*, int>* allocation) {
RTC_DCHECK_EQ(allocation->size(), allocatable_tracks.size());
// 統計所有的流
std::multimap<uint32_t, const AllocatableTrack*> list_max_bitrates;
for (const auto& observer_config : allocatable_tracks) {
if (include_zero_allocations ||
allocation->at(observer_config.observer) != 0) {
list_max_bitrates.insert(
{observer_config.config.max_bitrate_bps, &observer_config});
}
}
auto it = list_max_bitrates.begin();
while (it != list_max_bitrates.end()) {
RTC_DCHECK_GT(bitrate, 0);
// 對剩余碼率進行均分
uint32_t extra_allocation =
bitrate / static_cast<uint32_t>(list_max_bitrates.size());
// 求得均分后得碼率
uint32_t total_allocation =
extra_allocation + allocation->at(it->second->observer);
bitrate -= extra_allocation;
if (total_allocation > max_multiplier * it->first) {
// There is more than we can fit for this observer, carry over to the
// remaining observers.
// 如果均分后得碼率大於最大碼率得兩倍，則最多給某個流分配兩倍得最大碼率
bitrate += total_allocation - max_multiplier * it->first;
total_allocation = max_multiplier * it->first;
}
// Finally, update the allocation for this observer.
allocation->at(it->second->observer) = total_allocation;
it = list_max_bitrates.erase(it);
}
}
MaxRateAllocation()中:
- 首先將碼率滿足每個流的最大碼率
- 然后將多出來的碼率調用DistributeBitrateEvenly()均分給所有流，但同時不能超過流最大碼率的兩倍
2.5.2 視頻流碼率分配更新
當從BitratteAllocator中計算出每個stream的碼率之后，視頻流的碼率會通過VideoSendStreamImpl::OnBitrateUpdated()進行更新分配
uint32_t VideoSendStreamImpl::OnBitrateUpdated(BitrateAllocationUpdate update) {
RTC_DCHECK_RUN_ON(worker_queue_);
RTC_DCHECK(rtp_video_sender_->IsActive())
<< "VideoSendStream::Start has not been called.";
// When the BWE algorithm doesn't pass a stable estimate, we'll use the
// unstable one instead.
if (update.stable_target_bitrate.IsZero()) {
update.stable_target_bitrate = update.target_bitrate;
}
// 通過目標碼率，計算編碼碼率，保護碼率(fec, rtx等)分配
rtp_video_sender_->OnBitrateUpdated(update, stats_proxy_->GetSendFrameRate());
// 獲取更新后的目標編碼碼率
encoder_target_rate_bps_ = rtp_video_sender_->GetPayloadBitrateBps();
// 獲取更新后的目標fec碼率
const uint32_t protection_bitrate_bps =
rtp_video_sender_->GetProtectionBitrateBps();
// 計算link_allocation
DataRate link_allocation = DataRate::Zero();
if (encoder_target_rate_bps_ > protection_bitrate_bps) {
link_allocation =
DataRate::BitsPerSec(encoder_target_rate_bps_ - protection_bitrate_bps);//???
}
// 計算overhead(一些rtp頭什么的)
DataRate overhead =
update.target_bitrate - DataRate::BitsPerSec(encoder_target_rate_bps_);
DataRate encoder_stable_target_rate = update.stable_target_bitrate;
if (encoder_stable_target_rate > overhead) {
// 保守碼率比overhead大，則保守編碼碼率為保守碼率 - overhead
encoder_stable_target_rate = encoder_stable_target_rate - overhead;
} else {
// 否則，保守編碼碼率直接設置為保守碼率
encoder_stable_target_rate = DataRate::BitsPerSec(encoder_target_rate_bps_);
}
// encoder_target_rate_bps_不得大於上限
encoder_target_rate_bps_ =
std::min(encoder_max_bitrate_bps_, encoder_target_rate_bps_);
encoder_stable_target_rate =
std::min(DataRate::BitsPerSec(encoder_max_bitrate_bps_),
encoder_stable_target_rate);
DataRate encoder_target_rate = DataRate::BitsPerSec(encoder_target_rate_bps_);
//link_allocation不小於encoder_target_rate
link_allocation = std::max(encoder_target_rate, link_allocation);
// 更新編碼器
video_stream_encoder_->OnBitrateUpdated(
encoder_target_rate, encoder_stable_target_rate, link_allocation,
rtc::dchecked_cast<uint8_t>(update.packet_loss_ratio * 256),
update.round_trip_time.ms(), update.cwnd_reduce_ratio);
stats_proxy_->OnSetEncoderTargetRate(encoder_target_rate_bps_);
return protection_bitrate_bps;
}
VideoSendStreamImpl::OnBitrateUpdated()中：
-
首先使用RtpVideoSender::OnBitrateUpdated()，通過分配給該流的目標碼率，計算分配的編碼碼率和抗丟包碼率，將在2.5.2.1進行詳述
-
然后獲取目標編碼碼率(encoder_target_rate_bps)和目標保護碼率(protection_bitrate_bps，分配給fec, nack等), 並且定義了一個變量link_allocation，這個值表示的是當前video stream能使用的網絡通道容量（已排除調protection rate），這個值最低會比encoder_target_rate大。但是encoder_target_rate_bps_ - protection_bitrate_bps這一步的計算無法理解，因為encoder_target_rate_bps已經排除了protection_bitrate_bps，很費解。
-
最終會調用VideoStreamEncoder::OnBitrateUpdated()，使用新的編碼碼率去更新編碼器編碼碼率，調整各simulcast層級和temporal layer的碼率以及幀率，在2.5.2.2進行詳述
2.5.2.1 編碼碼率和抗丟包碼率的更新
RtpVideoSender::OnBitrateUpdated()中會通過分配給當前流的目標碼率和當前的幀率計算新的編碼碼率和抗丟包碼率
void RtpVideoSender::OnBitrateUpdated(BitrateAllocationUpdate update,
int framerate) {
// Substract overhead from bitrate.
// 計算simulcast流的平均overhead, overhead:Rtp擴展頭 + rtp頭開銷
MutexLock lock(&mutex_);
size_t num_active_streams = 0;
size_t overhead_bytes_per_packet = 0;
for (const auto& stream : rtp_streams_) {
if (stream.rtp_rtcp->SendingMedia()) {
// 統計rtp上的擴展頭+rtp頭開銷
overhead_bytes_per_packet += stream.rtp_rtcp->ExpectedPerPacketOverhead();
++num_active_streams;
}
}
if (num_active_streams > 1) {
overhead_bytes_per_packet /= num_active_streams;
}
// media_packet_header + udp header
// 計算每個包的over_head
DataSize packet_overhead = DataSize::Bytes(
overhead_bytes_per_packet + transport_overhead_bytes_per_packet_);
// 計算包的最大大小
DataSize max_total_packet_size = DataSize::Bytes(
rtp_config_.max_packet_size + transport_overhead_bytes_per_packet_);
uint32_t payload_bitrate_bps = update.target_bitrate.bps();
if (send_side_bwe_with_overhead_ && has_packet_feedback_) {
// 如果預估碼率是包含over_head_的，則減去overhead這部分
DataRate overhead_rate =
CalculateOverheadRate(update.target_bitrate, max_total_packet_size,
packet_overhead, Frequency::Hertz(framerate));
// TODO(srte): We probably should not accept 0 payload bitrate here.
// 計算payload的碼率
payload_bitrate_bps = rtc::saturated_cast<uint32_t>(payload_bitrate_bps -
overhead_rate.bps());
}
// Get the encoder target rate. It is the estimated network rate -
// protection overhead.
// TODO(srte): We should multiply with 255 here.
// 此時的payload_bitrate_bps還是包括fec的，要減去fec部分才能是編碼碼率
// fec_controller根據目標碼率和丟包率，更新fec_rate，同時計算fec需要使用的碼率
// 和剩下的給編碼器的碼率
encoder_target_rate_bps_ = fec_controller_->UpdateFecRates(
payload_bitrate_bps, framerate,
rtc::saturated_cast<uint8_t>(update.packet_loss_ratio * 256),
loss_mask_vector_, update.round_trip_time.ms());
if (!fec_allowed_) {
// 沒開啟fec，直接設置為encoder_target_bitrate
encoder_target_rate_bps_ = payload_bitrate_bps;
// fec_controller_->UpdateFecRates() was still called so as to allow
// |fec_controller_| to update whatever internal state it might have,
// since |fec_allowed_| may be toggled back on at any moment.
}
// Subtract packetization overhead from the encoder target. If target rate
// is really low, cap the overhead at 50%. This also avoids the case where
// |encoder_target_rate_bps_| is 0 due to encoder pause event while the
// packetization rate is positive since packets are still flowing.
// 如果目標碼率實在太低，也要把payload打包的開銷壓在50%以下，payload打包開銷
// 主要是發生在h264和av1，h264在做聚合包的時候需要添加nal進行分割等，詳情見RtpPacketizerH264::NextPacket()
// 這是為了防止在編碼器暫停時仍在打包而導致的目標編碼碼率被設置成0的情況,
uint32_t packetization_rate_bps =
std::min(GetPacketizationOverheadRate(), encoder_target_rate_bps_ / 2);
// 編碼碼率減去rtp打包頭
encoder_target_rate_bps_ -= packetization_rate_bps;
loss_mask_vector_.clear();
uint32_t encoder_overhead_rate_bps = 0;
if (send_side_bwe_with_overhead_ && has_packet_feedback_) {
// TODO(srte): The packet size should probably be the same as in the
// CalculateOverheadRate call above (just max_total_packet_size), it doesn't
// make sense to use different packet rates for different overhead
// calculations.
// 通過encoder rate重新計算overhead
DataRate encoder_overhead_rate = CalculateOverheadRate(
DataRate::BitsPerSec(encoder_target_rate_bps_),
max_total_packet_size - DataSize::Bytes(overhead_bytes_per_packet),
packet_overhead, Frequency::Hertz(framerate));
encoder_overhead_rate_bps = std::min(
encoder_overhead_rate.bps<uint32_t>(),
update.target_bitrate.bps<uint32_t>() - encoder_target_rate_bps_);
}
// When the field trial "WebRTC-SendSideBwe-WithOverhead" is enabled
// protection_bitrate includes overhead.
const uint32_t media_rate = encoder_target_rate_bps_ +
encoder_overhead_rate_bps +
packetization_rate_bps;
RTC_DCHECK_GE(update.target_bitrate, DataRate::BitsPerSec(media_rate));
protection_bitrate_bps_ = update.target_bitrate.bps() - media_rate;
}
RtpVideoSender::OnBitrateUpdated()中:
-
整體的碼率構成是: 分配的總碼率 = 媒體碼率 + 保護碼率
媒體碼率 = 目標編碼碼率 + 媒體打包碼率 + 封裝(rtp頭等)
保護碼率 = fec碼率 + nack_rate(重傳碼率)
-
首先計算當前stream下所有stream(simulcast)的平均overhead，這個平均overhead指的是rtp擴展頭+rtp頭
-
然后計算每個包的開銷 packet_overhead, 這個開銷是在rtp擴展頭+rtp頭的基礎上增加了transport層的開銷(srtp, udp)
-
如果預估的碼率已經考慮了overhead(send_side_bwe_with_overhead)，則將目標碼率減去overhead_rate，得到payload_bitrate_bps(有效負載碼率)
-
將payload_bitrate_bps(有效負載碼率)， 幀率，丟包率，fec掩碼表，rtt傳到FecControllerDefault::UpdateFecRates()中，通過FecController重新計算分配FEC和Nack等的碼率，更新fec_rate，同時根據更新后的的video_rate和fec_rate, nack_rate的比例，計算出當前payload_bitrate_bps下的新的有效載荷碼率
-
接着把載荷碼率減去媒體打包開銷(H264中的NALU，或者聚合包等會有額外的打包開銷，不是直接)，就得到目標編碼碼率了
-
最終通過修正過的編碼碼率和當前幀率和包大小，使用CalculateOverheadRate()重新計算了一次overhead，然后把encoder_target_rate_bps（目標編碼碼率） + encoder_overhead_rate_bps (修正過的overhead,rtp頭的) + packetization_rate_bps(打包開銷) 就得到media_rate(媒體的碼率), 將目標編碼碼率減去media_rate就得到保護碼率(protection_bitrate_bps)，
整個過程可以總結為:
目標碼率減去平均額外開銷得到有效負載碼率
有效負載碼率經過FEC得到包含了打包的目標編碼碼率
打包的目標編碼碼率減去打包開銷得到目標編碼碼率
基於目標編碼碼率和包平均大小，幀率重新計算overhead，將編碼碼率 + overhead + 打包開銷就得到了媒體碼率(media_rate)
目標碼率 - 媒體碼率(meida_rate) 得到保護碼率(protection_bitrate_bps)
其中FecControllerDefault::UpdateFecRates()計算有效載荷碼率的詳情如下:
uint32_t FecControllerDefault::UpdateFecRates(
uint32_t estimated_bitrate_bps,
int actual_framerate_fps,
uint8_t fraction_lost,
std::vector<bool> loss_mask_vector,
int64_t round_trip_time_ms) {
float target_bitrate_kbps =
static_cast<float>(estimated_bitrate_bps) / 1000.0f;
// Sanity check.
if (actual_framerate_fps < 1.0) {
actual_framerate_fps = 1.0;
}
// 通過丟包率，碼率等更新I幀和P幀的fec_rate
FecProtectionParams delta_fec_params;
FecProtectionParams key_fec_params;
{
MutexLock lock(&mutex_);
loss_prot_logic_->UpdateBitRate(target_bitrate_kbps);
loss_prot_logic_->UpdateRtt(round_trip_time_ms);
// Update frame rate for the loss protection logic class: frame rate should
// be the actual/sent rate.
loss_prot_logic_->UpdateFrameRate(actual_framerate_fps);
// Returns the filtered packet loss, used for the protection setting.
// The filtered loss may be the received loss (no filter), or some
// filtered value (average or max window filter).
// Use max window filter for now.
media_optimization::FilterPacketLossMode filter_mode =
media_optimization::kMaxFilter;
uint8_t packet_loss_enc = loss_prot_logic_->FilteredLoss(
clock_->TimeInMilliseconds(), filter_mode, fraction_lost);
// For now use the filtered loss for computing the robustness settings.
loss_prot_logic_->UpdateFilteredLossPr(packet_loss_enc);
if (loss_prot_logic_->SelectedType() == media_optimization::kNone) {
return estimated_bitrate_bps;
}
// Update method will compute the robustness settings for the given
// protection method and the overhead cost
// the protection method is set by the user via SetVideoProtection.
loss_prot_logic_->UpdateMethod();
// Get the bit cost of protection method, based on the amount of
// overhead data actually transmitted (including headers) the last
// second.
// Get the FEC code rate for Key frames (set to 0 when NA).
key_fec_params.fec_rate =
loss_prot_logic_->SelectedMethod()->RequiredProtectionFactorK();
// Get the FEC code rate for Delta frames (set to 0 when NA).
delta_fec_params.fec_rate =
loss_prot_logic_->SelectedMethod()->RequiredProtectionFactorD();
// The RTP module currently requires the same |max_fec_frames| for both
// key and delta frames.
delta_fec_params.max_fec_frames =
loss_prot_logic_->SelectedMethod()->MaxFramesFec();
key_fec_params.max_fec_frames =
loss_prot_logic_->SelectedMethod()->MaxFramesFec();
}
// Set the FEC packet mask type. |kFecMaskBursty| is more effective for
// consecutive losses and little/no packet re-ordering. As we currently
// do not have feedback data on the degree of correlated losses and packet
// re-ordering, we keep default setting to |kFecMaskRandom| for now.
delta_fec_params.fec_mask_type = kFecMaskRandom;
key_fec_params.fec_mask_type = kFecMaskRandom;
// Update protection callback with protection settings.
uint32_t sent_video_rate_bps = 0;
uint32_t sent_nack_rate_bps = 0;
uint32_t sent_fec_rate_bps = 0;
// Rate cost of the protection methods.
float protection_overhead_rate = 0.0f;
// TODO(Marco): Pass FEC protection values per layer.
// 將nack，fec的碼率更新,
// 獲取更新后的sent_video_rate_bps(視頻發送碼率), nack_rate(nack碼率)
// fec_rate(fec碼率)
protection_callback_->ProtectionRequest(
&delta_fec_params, &key_fec_params, &sent_video_rate_bps,
&sent_nack_rate_bps, &sent_fec_rate_bps);
// 計算總發送碼率
uint32_t sent_total_rate_bps =
sent_video_rate_bps + sent_nack_rate_bps + sent_fec_rate_bps;
// Estimate the overhead costs of the next second as staying the same
// wrt the source bitrate.
if (sent_total_rate_bps > 0) {
// 計算protection_rate
protection_overhead_rate =
static_cast<float>(sent_nack_rate_bps + sent_fec_rate_bps) /
sent_total_rate_bps;
}
// Cap the overhead estimate to a threshold, default is 50%.
protection_overhead_rate =
std::min(protection_overhead_rate, overhead_threshold_);
// Source coding rate: total rate - protection overhead.
// 計算media_rate
return estimated_bitrate_bps * (1.0 - protection_overhead_rate);
}
在FecControllerDefault::UpdateFecRates()中
-
首先會將丟包率， 幀率等輸入到loss_prot_logic，其最終會生成給關鍵幀和非關鍵幀的fec_rate，fec調整這一塊可參見webrtc源碼分析-fec
-
在獲取更新完成后的碼率后調用RtpVideoSender::ProtectionRequest()進行計算更新完成后的video_rate, fec_rate, nack_rate
int RtpVideoSender::ProtectionRequest(const FecProtectionParams* delta_params, const FecProtectionParams* key_params, uint32_t* sent_video_rate_bps, uint32_t* sent_nack_rate_bps, uint32_t* sent_fec_rate_bps) { *sent_video_rate_bps = 0; *sent_nack_rate_bps = 0; *sent_fec_rate_bps = 0; for (const RtpStreamSender& stream : rtp_streams_) { // 設置該rtp_rtcp上的fec_rate stream.rtp_rtcp->SetFecProtectionParams(*delta_params, *key_params); // 統計設置完之后的video_rate, fec_rate, nack_rate auto send_bitrate = stream.rtp_rtcp->GetSendRates(); *sent_video_rate_bps += send_bitrate[RtpPacketMediaType::kVideo].bps(); *sent_fec_rate_bps += send_bitrate[RtpPacketMediaType::kForwardErrorCorrection].bps(); *sent_nack_rate_bps += send_bitrate[RtpPacketMediaType::kRetransmission].bps(); } return 0; } -
基於更新后的fec_rate和nack_rate, 計算protection_rate(fec_rate加nack_rate)，protection_rate會被限制在50%以內，最后，通過estimated_bitrate_bps * (1- protection_rate)得到video_bitrate,
而使用CalculateOverheadRate()計算overhead的方式如下，原理上很簡單，通過目標碼率/包大小得到包數量，然后將包數量 * overhead就得到overhead了
/**
* @description: 求overhead的每秒碼率，計算方式是通過目標碼率和幀率得到幀大小
* 計算的方式很奇怪：data_rate / framerate / packet_size * framerate * overhead_per_packet
* 明顯是可以看到中間的framerate可以消掉更符合邏輯： data_rate / packet_size * overhead_per_packet, 難道只是為了中間的ceil()操作？
* @param {data_rate} 分配的碼率
* @param {packet_size} 包的大小
* @param {overhead_per_packet} 每個包的overhead
* @param {framerate} 當前幀率
* @return {*}
*/
DataRate RtpVideoSender::CalculateOverheadRate(DataRate data_rate,
DataSize packet_size,
DataSize overhead_per_packet,
Frequency framerate) const {
Frequency packet_rate = data_rate / packet_size;
if (use_frame_rate_for_overhead_) {
framerate = std::max(framerate, Frequency::Hertz(1));// 獲取幀率
DataSize frame_size = data_rate / framerate; // 計算幀大小
int packets_per_frame = ceil(frame_size / packet_size); // 計算每幀有多少包
packet_rate = packets_per_frame * framerate; // 計算包率
}
return packet_rate.RoundUpTo(Frequency::Hertz(1)) * overhead_per_packet;
}
2.5.2.2 更新編碼器
VideoStream的分配的碼率計算出來后，編碼器的會通過VideoStreamEncoder::OnBitrateUpdated()進行編碼碼率的分配更新，會重新調整simulcast 和 temporal layer各層級的碼率，以及幀率大小。
/**
* @description: 更新編碼碼率，更新當前stream下分配給各個simulcast和
* temporal layer的碼率，調整幀率
* @param {target_bitrate} 目標編碼碼率
* @param {stable_target_bitrate} 目標保守編碼碼率
* @param {link_allocation} 能使用的網絡通道容量
* @param {fraction_lost} 丟包率
* @param {round_trip_time_ms} rtt
* @param {cwnd_reduce_ratio}
* @return {*}
*/
void VideoStreamEncoder::OnBitrateUpdated(DataRate target_bitrate,
DataRate stable_target_bitrate,
DataRate link_allocation,
uint8_t fraction_lost,
int64_t round_trip_time_ms,
double cwnd_reduce_ratio) {
// 線程檢查
RTC_DCHECK_GE(link_allocation, target_bitrate);
if (!encoder_queue_.IsCurrent()) {
encoder_queue_.PostTask([this, target_bitrate, stable_target_bitrate,
link_allocation, fraction_lost, round_trip_time_ms,
cwnd_reduce_ratio] {
DataRate updated_target_bitrate =
UpdateTargetBitrate(target_bitrate, cwnd_reduce_ratio);
OnBitrateUpdated(updated_target_bitrate, stable_target_bitrate,
link_allocation, fraction_lost, round_trip_time_ms,
cwnd_reduce_ratio);
});
return;
}
RTC_DCHECK_RUN_ON(&encoder_queue_);
// target_bitrate == 0 -> video suspended(暫停)
const bool video_is_suspended = target_bitrate == DataRate::Zero();
const bool video_suspension_changed = video_is_suspended != EncoderPaused();
if (!video_is_suspended && settings_.encoder_switch_request_callback &&
encoder_selector_) {
// 竟然還能根據鏈路容量進行編碼器的切換，但是目前沒在其中看到encoder_selector的實現
if (auto encoder = encoder_selector_->OnAvailableBitrate(link_allocation)) {
QueueRequestEncoderSwitch(*encoder);
}
}
RTC_DCHECK(sink_) << "sink_ must be set before the encoder is active.";
RTC_LOG(LS_VERBOSE) << "OnBitrateUpdated, bitrate " << target_bitrate.bps()
<< " stable bitrate = " << stable_target_bitrate.bps()
<< " link allocation bitrate = " << link_allocation.bps()
<< " packet loss " << static_cast<int>(fraction_lost)
<< " rtt " << round_trip_time_ms;
if (encoder_) {
// 丟包率和rtt更新到encoder中並沒有實際使用
encoder_->OnPacketLossRateUpdate(static_cast<float>(fraction_lost) / 256.f);
encoder_->OnRttUpdate(round_trip_time_ms);
}
uint32_t framerate_fps = GetInputFramerateFps();
frame_dropper_.SetRates((target_bitrate.bps() + 500) / 1000, framerate_fps);
// 根據目標碼率設置到不同的layer中，然后設置到編碼器中，並告訴rtp_video_sender
EncoderRateSettings new_rate_settings{
VideoBitrateAllocation(), static_cast<double>(framerate_fps),
link_allocation, target_bitrate, stable_target_bitrate};
SetEncoderRates(UpdateBitrateAllocation(new_rate_settings));
if (target_bitrate.bps() != 0)
encoder_target_bitrate_bps_ = target_bitrate.bps();
stream_resource_manager_.SetTargetBitrate(target_bitrate);
if (video_suspension_changed) {
RTC_LOG(LS_INFO) << "Video suspend state changed to: "
<< (video_is_suspended ? "suspended" : "not suspended");
encoder_stats_observer_->OnSuspendChange(video_is_suspended);
}
if (video_suspension_changed && !video_is_suspended && pending_frame_ &&
!DropDueToSize(pending_frame_->size())) {
// 此時碼率可能，不再suspended了, 把pending_frame放入編碼隊列
int64_t pending_time_us =
clock_->CurrentTime().us() - pending_frame_post_time_us_;
if (pending_time_us < kPendingFrameTimeoutMs * 1000)
EncodeVideoFrame(*pending_frame_, pending_frame_post_time_us_);
pending_frame_.reset();
}
}
VideoStreamEncoder::OnBitrateUpdated()中:
- 首先進行執行任務線程檢查是否為encoder_queue，如果不在，則切換運行到該線程
- 根據link_allocation做編碼器的切換，在windows端沒有找到實現，源碼中有ios和安卓的實現，和移動端平台的編碼器有關系，目前還未深入了解，創建完成的編碼器會調用QueueRequestEncoderSwitch()對stream的編碼器重新創建初始化，並修改相關編碼參數，從這里能看出，webrtc是能夠支持中途變更編碼器的
- 把丟包率和rtt放到編碼器中，可能是想依靠他們做什么碼率控制，但目前沒有實現，OnPacketLossRateUpdate()和OnRttUpdate()兩個都是空函數
- 調用GetInputFramerateFps()獲取當前幀率，把當前的幀率和碼率放到frame_dropper中，讓frame_dropper在MaybeEncodeVideoFrame()進行幀編碼的時候計算是否需要丟幀以適應當前的碼率。
- 調用UpdateBitrateAllocation()，計算分配給當前stream下的simulcast和temporal的碼率，使用SetEncoderRates()將新分配的碼率設置到編碼器，同時告知下層RTCP模塊，可能發生了layer或者分辨率的變化，讓其發送RTCP報文告知對方
- 檢查在更新碼率后是否從suspend(分配碼率無法達到min_bitrate而導致的掛起狀態)狀態脫離，如果是，則將當前因為suspend而Pending 的Frame送入編碼
其中，將碼率分配到不同的simulcast和temporal layer的UpdateBitrateAllocation()見下：
VideoStreamEncoder::EncoderRateSettings
VideoStreamEncoder::UpdateBitrateAllocation(
const EncoderRateSettings& rate_settings) {
VideoBitrateAllocation new_allocation;
// Only call allocators if bitrate > 0 (ie, not suspended), otherwise they
// might cap the bitrate to the min bitrate configured.
// 通過rate_allocator重新分配碼率
if (rate_allocator_ && rate_settings.encoder_target > DataRate::Zero()) {
// 重新計算分配到simulcast和temporal下的碼率
// -> SimulcastRateAllocator::Allocate()
new_allocation = rate_allocator_->Allocate(VideoBitrateAllocationParameters(
rate_settings.encoder_target, rate_settings.stable_encoder_target,
rate_settings.rate_control.framerate_fps));
}
EncoderRateSettings new_rate_settings = rate_settings;
new_rate_settings.rate_control.target_bitrate = new_allocation;
new_rate_settings.rate_control.bitrate = new_allocation;
// VideoBitrateAllocator subclasses may allocate a bitrate higher than the
// target in order to sustain the min bitrate of the video codec. In this
// case, make sure the bandwidth allocation is at least equal the allocation
// as that is part of the document contract for that field.
new_rate_settings.rate_control.bandwidth_allocation =
std::max(new_rate_settings.rate_control.bandwidth_allocation,
DataRate::BitsPerSec(
new_rate_settings.rate_control.bitrate.get_sum_bps()));
if (bitrate_adjuster_) {
// 對碼率進行再調節，以平滑碼率調整的過程
VideoBitrateAllocation adjusted_allocation =
bitrate_adjuster_->AdjustRateAllocation(new_rate_settings.rate_control);
RTC_LOG(LS_VERBOSE) << "Adjusting allocation, fps = "
<< rate_settings.rate_control.framerate_fps << ", from "
<< new_allocation.ToString() << ", to "
<< adjusted_allocation.ToString();
new_rate_settings.rate_control.bitrate = adjusted_allocation;
}
return new_rate_settings;
}
UpdateBitrateAllocation()中:
-
調用rate_allocator_->Allocate()分配碼率到simulcast 和 temporal layer，實則調用的為SimulcastRateAllocator::Allocate(), simulcast的碼率分配策略是，先滿足max_bitrate的最低的layer的min_bitrate，如果有剩余的碼率，從低到高去滿足他們的target_bitrate;
VideoBitrateAllocation SimulcastRateAllocator::Allocate( VideoBitrateAllocationParameters parameters) { VideoBitrateAllocation allocated_bitrates; DataRate stable_rate = parameters.total_bitrate; if (stable_rate_settings_.IsEnabled() && parameters.stable_bitrate > DataRate::Zero()) { stable_rate = std::min(parameters.stable_bitrate, parameters.total_bitrate); } // 將碼率分配到不同的simulcast DistributeAllocationToSimulcastLayers(parameters.total_bitrate, stable_rate, &allocated_bitrates); // 通過分配好的simulcat碼率再分配到不同的temporalayers DistributeAllocationToTemporalLayers(&allocated_bitrates); return allocated_bitrates; }而分配給temporal的碼率，主要是根據SimulcastRateAllocator::GetTemporalRateAllocation()函數中選取的模型，一種是BaseHeavy, 對應的3個temporal 的值是(60%, 20%, 20%), 而另一種則是變化層數的，見下kLayerRateAllocation
static const float kLayerRateAllocation[kMaxTemporalStreams][kMaxTemporalStreams] = { {1.0f, 1.0f, 1.0f, 1.0f}, // 1 layer {0.6f, 1.0f, 1.0f, 1.0f}, // 2 layers {60%, 40%} {0.4f, 0.6f, 1.0f, 1.0f}, // 3 layers {40%, 20%, 40%} {0.25f, 0.4f, 0.6f, 1.0f} // 4 layers {25%, 15%, 20%, 40%} }; static const float kBaseHeavy3TlRateAllocation[kMaxTemporalStreams] = { 0.6f, 0.8f, 1.0f, 1.0f // 3 layers {60%, 20%, 20%} }; float SimulcastRateAllocator::GetTemporalRateAllocation( int num_layers, int temporal_id, bool base_heavy_tl3_alloc) { if (num_layers == 3 && base_heavy_tl3_alloc) { // 層數為3，並且選擇了heavy_base return kBaseHeavy3TlRateAllocation[temporal_id]; } return kLayerRateAllocation[num_layers - 1][temporal_id]; } -
調用bitrate_adjuster_->AdjustRateAllocation()使上面分配后的碼率趨向於平滑
用以設置新的碼率SetEncoderRates()如下:
void VideoStreamEncoder::SetEncoderRates(
const EncoderRateSettings& rate_settings) {
RTC_DCHECK_GT(rate_settings.rate_control.framerate_fps, 0.0);
// 檢測(fps, bitrate)是否發生改變
bool rate_control_changed =
(!last_encoder_rate_settings_.has_value() ||
last_encoder_rate_settings_->rate_control != rate_settings.rate_control);
// For layer allocation signal we care only about the target bitrate (not the
// adjusted one) and the target fps.
// 對於layer, 只是考慮target_bitrate和framerate_fps
bool layer_allocation_changed =
!last_encoder_rate_settings_.has_value() ||
last_encoder_rate_settings_->rate_control.target_bitrate !=
rate_settings.rate_control.target_bitrate ||
last_encoder_rate_settings_->rate_control.framerate_fps !=
rate_settings.rate_control.framerate_fps;
if (last_encoder_rate_settings_ != rate_settings) {
last_encoder_rate_settings_ = rate_settings;
}
if (!encoder_) {
return;
}
// |bitrate_allocation| is 0 it means that the network is down or the send
// pacer is full. We currently only report this if the encoder has an internal
// source. If the encoder does not have an internal source, higher levels
// are expected to not call AddVideoFrame. We do this since its unclear
// how current encoder implementations behave when given a zero target
// bitrate.
// TODO(perkj): Make sure all known encoder implementations handle zero
// target bitrate and remove this check.
if (!HasInternalSource() &&
rate_settings.rate_control.bitrate.get_sum_bps() == 0) {
return;
}
//fps和bitrate發生了改變，告知編碼器
if (rate_control_changed) {
// EncoderSimulcastProxy::SetRates()
// ->H264EncoderImpl::SetRates()
encoder_->SetRates(rate_settings.rate_control);
encoder_stats_observer_->OnBitrateAllocationUpdated(
send_codec_, rate_settings.rate_control.bitrate);
frame_encode_metadata_writer_.OnSetRates(
rate_settings.rate_control.bitrate,
static_cast<uint32_t>(rate_settings.rate_control.framerate_fps + 0.5));
stream_resource_manager_.SetEncoderRates(rate_settings.rate_control);
// 不同layer的流的碼率發生了變化，告知rtp_rtcp，在發送關鍵幀的時候，通過extension把最新的layer
// 分辨率帶上，詳見RtpSenderVideo::SetVideoLayersAllocation()和RtpSenderVideo::SetVideoLayersAllocationInternal()
if (layer_allocation_changed &&
allocation_cb_type_ ==
BitrateAllocationCallbackType::kVideoLayersAllocation) {
sink_->OnVideoLayersAllocationUpdated(CreateVideoLayersAllocation(
send_codec_, rate_settings.rate_control, encoder_->GetEncoderInfo()));
}
}
// 通知rtp_video_sender，由它通知rtp_rtcp等模塊，碼率是否發生了改變
if ((allocation_cb_type_ ==
BitrateAllocationCallbackType::kVideoBitrateAllocation) ||
(encoder_config_.content_type ==
VideoEncoderConfig::ContentType::kScreen &&
allocation_cb_type_ == BitrateAllocationCallbackType::
kVideoBitrateAllocationWhenScreenSharing)) {
// rtp_rtcp檢測到layer發生了變化后
sink_->OnBitrateAllocationUpdated(
// Update allocation according to info from encoder. An encoder may
// choose to not use all layers due to for example HW.
UpdateAllocationFromEncoderInfo(
rate_settings.rate_control.target_bitrate,
encoder_->GetEncoderInfo()));
}
}
VideoStreamEncoder::SetEncoderRates()中：
-
首先檢測FPS和Bitrate是否發生改變，如果是則告知編碼器
-
不同Layer的流的碼率發生了變化，告知RTP_RTCP，RtpSenderVideo有可能通過RTP-EXTENSION將Layer的碼率帶上
-
告知RTP_RTCP Layer的變化，層數可能發生改變，要發送相應的RTCP-XR
void RTCPSender::SetVideoBitrateAllocation( const VideoBitrateAllocation& bitrate) { MutexLock lock(&mutex_rtcp_sender_); // Check if this allocation is first ever, or has a different set of // spatial/temporal layers signaled and enabled, if so trigger an rtcp report // as soon as possible. // 檢測這個碼流分配器是否包含了新的時域/空域層，有的話盡快發送xr-rtcp告知對端 absl::optional<VideoBitrateAllocation> new_bitrate = CheckAndUpdateLayerStructure(bitrate); if (new_bitrate) { video_bitrate_allocation_ = *new_bitrate; RTC_LOG(LS_INFO) << "Emitting TargetBitrate XR for SSRC " << ssrc_ << " with new layers enabled/disabled: " << video_bitrate_allocation_.ToString(); next_time_to_send_rtcp_ = clock_->TimeInMilliseconds(); } else { video_bitrate_allocation_ = bitrate; } send_video_bitrate_allocation_ = true; SetFlag(kRtcpAnyExtendedReports, true); }這個XR-RTCP Report的內容詳見RFC3611， 也可參考RTCPSender::BuildExtendedReports()函數，可以看到包含了simulcast layer和temporal layer的碼率信息，這些信息可用於在服務端做simulcast和temporal轉發決策。
void RTCPSender::BuildExtendedReports(const RtcpContext& ctx, PacketSender& sender) { rtcp::ExtendedReports xr; xr.SetSenderSsrc(ssrc_); if (!sending_ && xr_send_receiver_reference_time_enabled_) { rtcp::Rrtr rrtr; rrtr.SetNtp(TimeMicrosToNtp(ctx.now_us_)); xr.SetRrtr(rrtr); } for (const rtcp::ReceiveTimeInfo& rti : ctx.feedback_state_.last_xr_rtis) { xr.AddDlrrItem(rti); } if (send_video_bitrate_allocation_) { rtcp::TargetBitrate target_bitrate; // 將每個simulcast layer和temporal layer的信息碼率提取出來放到report中 for (int sl = 0; sl < kMaxSpatialLayers; ++sl) { for (int tl = 0; tl < kMaxTemporalStreams; ++tl) { if (video_bitrate_allocation_.HasBitrate(sl, tl)) { target_bitrate.AddTargetBitrate( sl, tl, video_bitrate_allocation_.GetBitrate(sl, tl) / 1000); } } } xr.SetTargetBitrate(target_bitrate); send_video_bitrate_allocation_ = false; } sender.AppendPacket(xr); }
2.5.3 UpdateAllocationLimits()
BitrateAllocator::OnNetworkEstimateChanged()中將碼率分發各個視頻流中並更新編碼器碼率后，stream的limit(min_allocatable_rate, max_padding_rate,max_allocatable_rate)會發生改變，此時需要將變更后的limit重新統計，告知cc-controller變更后的limit，調用的是BitrateAllocator::UpdateAllocationLimits()
void BitrateAllocator::UpdateAllocationLimits() {
BitrateAllocationLimits limits;
for (const auto& config : allocatable_tracks_) {
uint32_t stream_padding = config.config.pad_up_bitrate_bps;
if (config.config.enforce_min_bitrate) {
limits.min_allocatable_rate +=
DataRate::BitsPerSec(config.config.min_bitrate_bps);
} else if (config.allocated_bitrate_bps == 0) {
stream_padding =
std::max(config.MinBitrateWithHysteresis(), stream_padding);
}
limits.max_padding_rate += DataRate::BitsPerSec(stream_padding);
limits.max_allocatable_rate +=
DataRate::BitsPerSec(config.config.max_bitrate_bps);
}
if (limits.min_allocatable_rate == current_limits_.min_allocatable_rate &&
limits.max_allocatable_rate == current_limits_.max_allocatable_rate &&
limits.max_padding_rate == current_limits_.max_padding_rate) {
return;
}
current_limits_ = limits;
RTC_LOG(LS_INFO) << "UpdateAllocationLimits : total_requested_min_bitrate: "
<< ToString(limits.min_allocatable_rate)
<< ", total_requested_padding_bitrate: "
<< ToString(limits.max_padding_rate)
<< ", total_requested_max_bitrate: "
<< ToString(limits.max_allocatable_rate);
limit_observer_->OnAllocationLimitsChanged(limits);
}
其中limit_observer_->OnAllocationLimitsChanged()會通過 RtpTransportControllerSend::UpdateStreamsConfig()會調用到cc-controller中
void RtpTransportControllerSend::UpdateStreamsConfig() {
streams_config_.at_time = Timestamp::Millis(clock_->TimeInMilliseconds());
if (controller_)
PostUpdates(controller_->OnStreamsConfig(streams_config_));
}
NetworkControlUpdate GoogCcNetworkController::OnStreamsConfig(
StreamsConfig msg) {
NetworkControlUpdate update;
if (msg.requests_alr_probing) {
probe_controller_->EnablePeriodicAlrProbing(*msg.requests_alr_probing);
}
if (msg.max_total_allocated_bitrate &&
*msg.max_total_allocated_bitrate != max_total_allocated_bitrate_) {
if (rate_control_settings_.TriggerProbeOnMaxAllocatedBitrateChange()) {
// 重設probe_controller的max_bitrate
update.probe_cluster_configs =
probe_controller_->OnMaxTotalAllocatedBitrate(
msg.max_total_allocated_bitrate->bps(), msg.at_time.ms());
} else {
probe_controller_->SetMaxBitrate(msg.max_total_allocated_bitrate->bps());
}
max_total_allocated_bitrate_ = *msg.max_total_allocated_bitrate;
}
bool pacing_changed = false;
if (msg.pacing_factor && *msg.pacing_factor != pacing_factor_) {
pacing_factor_ = *msg.pacing_factor;
pacing_changed = true;
}
if (msg.min_total_allocated_bitrate &&
*msg.min_total_allocated_bitrate != min_total_allocated_bitrate_) {
min_total_allocated_bitrate_ = *msg.min_total_allocated_bitrate;
pacing_changed = true;
if (use_min_allocatable_as_lower_bound_) {
ClampConstraints();
// 重設delay_based_bwe和bandwidth_estimation的limit
delay_based_bwe_->SetMinBitrate(min_data_rate_);
bandwidth_estimation_->SetMinMaxBitrate(min_data_rate_, max_data_rate_);
}
}
if (msg.max_padding_rate && *msg.max_padding_rate != max_padding_rate_) {
max_padding_rate_ = *msg.max_padding_rate;
pacing_changed = true;
}
if (pacing_changed)
update.pacer_config = GetPacingRates(msg.at_time);
return update;
}
在GoogCcNetworkController::OnStreamsConfig()中:
- 檢測到max_total_allocated_bitrate發生變更的時候，會調用probe_controller_->OnMaxTotalAllocatedBitrate()更新max_total_allocated_bitrate，同時獲取新的探測進行探測
- 重設delay_based_bwe和bandwidth_estimation的limit
- 檢測到limit發生改變的時候，更新update.pacer_config，重新設置pacing_rate
- 返回update，調用PostUpdates()從頭開始進行碼率調整