[時間：2016-11] [狀態：Open]
[關鍵詞：android，nuplayer，開源播放器，播放框架，渲染器，render]
0 導讀
之前我們分析了NuPlayer的實現代碼，本文將重點聚焦於其中的一部分——渲染器（Renderer）。
從功能安排來說，Renderer的主要功能有：
- 音視頻原始數據緩存操作
- 音頻播放（到聲卡）
- 視頻顯示（到顯卡）
- 音視頻同步
- 其他輔助播放器控制的操作
- 其他獲取渲染狀態/屬性的接口
接下來主要從Renderer的對外接口和實現說明下其中的處理邏輯。
本文是我的NuPlayer播放框架的第四篇。
1 NuPlayer::Renderer對外接口及主要成員
// code frome ~/frameworks/av/media/libmediaplayerservice/nuplayer/NuPlayerRenderer.h
struct NuPlayer::Renderer : public AHandler {
Renderer(const sp<MediaPlayerBase::AudioSink> &sink,
const sp<AMessage> ¬ify, uint32_t flags = 0);
static size_t AudioSinkCallback(MediaPlayerBase::AudioSink *audioSink,
void *data, size_t size, void *me,
MediaPlayerBase::AudioSink::cb_event_t event);
// 緩沖音視頻原始數據
void queueBuffer(bool audio,
const sp<ABuffer> &buffer, const sp<AMessage> ¬ifyConsumed);
void queueEOS(bool audio, status_t finalResult);
status_t setPlaybackSettings(const AudioPlaybackRate &rate /* sanitized */);
status_t getPlaybackSettings(AudioPlaybackRate *rate /* nonnull */);
status_t setSyncSettings(const AVSyncSettings &sync, float videoFpsHint);
status_t getSyncSettings(AVSyncSettings *sync /* nonnull */, float *videoFps /* nonnull */);
void flush(bool audio, bool notifyComplete);
void signalTimeDiscontinuity();
void signalAudioSinkChanged();
void signalDisableOffloadAudio();
void signalEnableOffloadAudio();
void pause();
void resume();
void setVideoFrameRate(float fps);
status_t getCurrentPosition(int64_t *mediaUs);
int64_t getVideoLateByUs();
status_t openAudioSink( const sp<AMessage> &format, bool offloadOnly,
bool hasVideo, uint32_t flags, bool *isOffloaded);
void closeAudioSink();
private:
struct QueueEntry {
sp<ABuffer> mBuffer;
sp<AMessage> mNotifyConsumed;
size_t mOffset;
status_t mFinalResult;
int32_t mBufferOrdinal;
};
static const int64_t kMinPositionUpdateDelayUs;
sp<MediaPlayerBase::AudioSink> mAudioSink;
bool mUseVirtualAudioSink;
sp<AMessage> mNotify;
Mutex mLock;
uint32_t mFlags;
List<QueueEntry> mAudioQueue; // 音頻緩沖
List<QueueEntry> mVideoQueue; // 視頻緩沖
uint32_t mNumFramesWritten;
sp<VideoFrameScheduler> mVideoScheduler;
sp<MediaClock> mMediaClock;
float mPlaybackRate; // audio track rate
}
首先看到的是Renderer本身是AHandler的子類。還記得之前的AHandler和ALooper配合使用的機制嘛？其中ALooper位於NuPlayer中，變量名為mRendererLooper。
2 NuPlayer中調用的Renderer接口
先回顧下NuPlayer源碼解析中的調用接口。
- 構造/析構函數
- 設置播放控制參數——setPlaybackSettings/getPlaybackSettings/setVideoFrameRate/setSyncSettings/getSyncSettings
- AudioSink相關——openAudioSink/closeAudioSink
- 控制接口——pause/flush/resume/queueEOS
- 音頻狀態更新——signalEnableOffloadAudio/signalDisableOffloadAudio
- 音視頻原始數據輸入——queueBuffer
NuPlayer中並未顯示調用，而是將Renderer設置給ADecoder使用
if (mVideoDecoder != NULL) {
mVideoDecoder->setRenderer(mRenderer);
}
if (mAudioDecoder != NULL) {
mAudioDecoder->setRenderer(mRenderer);
}
3 Renderer具體接口分析
構造函數喝析構函數
構造函數最主要的是創建一個MediaClock，用於同步和計時。主要代碼如下：
mMediaClock = new MediaClock;
mPlaybackRate = mPlaybackSettings.mSpeed;
mMediaClock->setPlaybackRate(mPlaybackRate);
由於AHandler是智能指針，可以不考慮析構函數。不過可以看下代碼中實現：
NuPlayer::Renderer::~Renderer() {
if (offloadingAudio()) {
mAudioSink->stop(); // 主要是針對AudioSink的處理
mAudioSink->flush();
mAudioSink->close();
}
}
設置播放控制參數類接口
音頻回放參數設置-setPlaybackSettings/getPlaybackSettings
主要接口定義及參數如下：
status_t setPlaybackSettings(const AudioPlaybackRate &rate /* sanitized */);
status_t getPlaybackSettings(AudioPlaybackRate *rate /* nonnull */);
struct AudioPlaybackRate {
float mSpeed; // 播放倍速
float mPitch; // 聲調參數
enum AudioTimestretchStretchMode mStretchMode; // 拉伸模式
enum AudioTimestretchFallbackMode mFallbackMode; // 備用模式
};
從實際接口含義來看主要控制音頻播放速率。最終設置函數將參數傳遞給mMediaClock->setPlaybackRate函數。
視頻播放幀率參數-setVideoFrameRate
函數原型如下：void setVideoFrameRate(float fps);。只有一個參數視頻播放幀率fps，最終實現函數將該參數設置給mVideoScheduler。實現如下：
void NuPlayer::Renderer::onSetVideoFrameRate(float fps) {
if (mVideoScheduler == NULL) {
mVideoScheduler = new VideoFrameScheduler();
}
mVideoScheduler->init(fps);
}
音視頻同步參數-setSyncSettings/getSyncSettings
接口聲明及主要參數如下：
status_t setSyncSettings(const AVSyncSettings &sync, float videoFpsHint);
status_t getSyncSettings(AVSyncSettings *sync /* nonnull */, float *videoFps /* nonnull */);
// from ~/frameworks/av/include/media/AVSyncSettings.h
struct AVSyncSettings {
AVSyncSource mSource; // 同步基准
AVSyncAudioAdjustMode mAudioAdjustMode; // 音頻調整方式
float mTolerance; // 最大容忍的調速時間
AVSyncSettings()
: mSource(AVSYNC_SOURCE_DEFAULT),
mAudioAdjustMode(AVSYNC_AUDIO_ADJUST_MODE_DEFAULT),
mTolerance(.044f) { }
};
看代碼實現就會發現，Renderer中並沒有實現setSyncSettings，只是判斷了必須使用必須使用默認的同步方式，判斷邏輯如下：
status_t NuPlayer::Renderer::onConfigSync(const AVSyncSettings &sync, float videoFpsHint __unused) {
if (sync.mSource != AVSYNC_SOURCE_DEFAULT) {
return BAD_VALUE;
}
// TODO: support sync sources
return INVALID_OPERATION;
}
至於這里涉及的MediaClock、AudioSink、VideoFrameSchedule后續有專門介紹。
AudioSink相關-openAudioSink/closeAudioSink
主要用於創建和關閉AudioSink，聲明如下：
status_t openAudioSink(
const sp<AMessage> &format,
bool offloadOnly,
bool hasVideo,
uint32_t flags,
bool *isOffloaded);
void closeAudioSink();
后續會解釋兩個接口。
控制接口-pause/flush/resume/queueEOS
pause/resume接口
暫停和恢復接口，實現類似，pause接口最終實現是在onPause中：
void NuPlayer::Renderer::onPause() {
if (mPaused) {
return;
}
{
Mutex::Autolock autoLock(mLock);
// we do not increment audio drain generation so that we fill audio buffer during pause.
++mVideoDrainGeneration;
prepareForMediaRenderingStart_l();
mPaused = true;
mMediaClock->setPlaybackRate(0.0); // 設置成0.0，后面解釋為什么
}
mDrainAudioQueuePending = false;
mDrainVideoQueuePending = false;
// Note: audio data may not have been decoded, and the AudioSink may not be opened.
mAudioSink->pause();
startAudioOffloadPauseTimeout();
}
其最終通過mMediaClock->setPlaybackRate和mAudioSink->pause接口實現暫停功能。
resume接口最終實現是在onResume中，代碼如下：
void NuPlayer::Renderer::onResume() {
if (!mPaused) {
return;
}
// Note: audio data may not have been decoded, and the AudioSink may not be opened.
cancelAudioOffloadPauseTimeout();
if (mAudioSink->ready()) {
status_t err = mAudioSink->start();
if (err != OK) {
ALOGE("cannot start AudioSink err %d", err);
notifyAudioTearDown(kDueToError);
}
}
{
Mutex::Autolock autoLock(mLock);
mPaused = false;
// rendering started message may have been delayed if we were paused.
if (mRenderingDataDelivered) {
notifyIfMediaRenderingStarted_l();
}
// configure audiosink as we did not do it when pausing
if (mAudioSink != NULL && mAudioSink->ready()) {
mAudioSink->setPlaybackRate(mPlaybackSettings);
}
mMediaClock->setPlaybackRate(mPlaybackRate);
if (!mAudioQueue.empty()) {
postDrainAudioQueue_l();
}
}
if (!mVideoQueue.empty()) {
postDrainVideoQueue();
}
}
基本上是通過mAudioSink->start()和mMediaClock->setPlaybackRate實現，這過程中也有音視頻隊列清空的操作。
flush接口
主要分為針對音頻的flush和針對視頻的flush，具體實現時，音頻主要是使用AudioSink的pause/flush/start接口，視頻主要是使用清空緩沖隊列和mVideoScheduler->restart實現。詳細實現建議參考NuPlayer::Renderer::onFlush的代碼。
queueEOS
添加流結束標志，最終實現是在onQueueEOS接口中，代碼如下：
void NuPlayer::Renderer::onQueueEOS(const sp<AMessage> &msg) {
int32_t audio;
CHECK(msg->findInt32("audio", &audio));
if (dropBufferIfStale(audio, msg)) {
return;
}
int32_t finalResult;
CHECK(msg->findInt32("finalResult", &finalResult));
QueueEntry entry;
entry.mOffset = 0;
entry.mFinalResult = finalResult;
if (audio) { // 音頻EOS
Mutex::Autolock autoLock(mLock);
if (mAudioQueue.empty() && mSyncQueues) {
syncQueuesDone_l();
}
mAudioQueue.push_back(entry);
postDrainAudioQueue_l();
} else { // 視頻EOS
if (mVideoQueue.empty() && getSyncQueues()) {
Mutex::Autolock autoLock(mLock);
syncQueuesDone_l();
}
mVideoQueue.push_back(entry);
postDrainVideoQueue();
}
}
音視頻原始數據輸入——queueBuffer
在NuPlayer中沒看到這個函數調用，但總體來說這個應該由音視頻解碼器調用，主要將解碼之后的音視頻原始數據通知顯示端並作緩存和同步。主要實現代碼如下：（有刪減）
void NuPlayer::Renderer::onQueueBuffer(const sp<AMessage> &msg) {
int32_t audio;
CHECK(msg->findInt32("audio", &audio));
if (dropBufferIfStale(audio, msg)) {
return;
}
sp<ABuffer> buffer;
CHECK(msg->findBuffer("buffer", &buffer)); // 傳入的數據存儲在這里
QueueEntry entry;
entry.mBuffer = buffer;
entry.mNotifyConsumed = notifyConsumed;
entry.mOffset = 0;
entry.mFinalResult = OK;
entry.mBufferOrdinal = ++mTotalBuffersQueued;
// 將數據放到音頻或者視頻緩沖隊列中
if (audio) {
Mutex::Autolock autoLock(mLock);
mAudioQueue.push_back(entry);
postDrainAudioQueue_l();
} else {
mVideoQueue.push_back(entry);
postDrainVideoQueue();
}
// 后續代碼是做同步的
Mutex::Autolock autoLock(mLock);
if (!mSyncQueues || mAudioQueue.empty() || mVideoQueue.empty()) {
return;
}
sp<ABuffer> firstAudioBuffer = (*mAudioQueue.begin()).mBuffer;
sp<ABuffer> firstVideoBuffer = (*mVideoQueue.begin()).mBuffer;
if (firstAudioBuffer == NULL || firstVideoBuffer == NULL) {
// EOS signalled on either queue.
syncQueuesDone_l();
return;
}
int64_t firstAudioTimeUs;
int64_t firstVideoTimeUs;
CHECK(firstAudioBuffer->meta()
->findInt64("timeUs", &firstAudioTimeUs));
CHECK(firstVideoBuffer->meta()
->findInt64("timeUs", &firstVideoTimeUs));
int64_t diff = firstVideoTimeUs - firstAudioTimeUs;
ALOGV("queueDiff = %.2f secs", diff / 1E6);
if (diff > 100000ll) { //
// Audio data starts More than 0.1 secs before video.
// Drop some audio.
(*mAudioQueue.begin()).mNotifyConsumed->post();
mAudioQueue.erase(mAudioQueue.begin());
return;
}
syncQueuesDone_l();
}
4 MediaClock簡介
看名字，MediaClock有點時鍾同步的感覺，說白了就是一個多媒體時鍾，是libstagefright提供的一個公共類。具體接口如下：
struct MediaClock : public RefBase {
MediaClock();
void setStartingTimeMedia(int64_t startingTimeMediaUs);
void clearAnchor();
void updateAnchor( int64_t anchorTimeMediaUs,
int64_t anchorTimeRealUs, int64_t maxTimeMediaUs = INT64_MAX);
void updateMaxTimeMedia(int64_t maxTimeMediaUs);
void setPlaybackRate(float rate);
float getPlaybackRate() const;
// 查詢與實際時間|realUs|對應的多媒體時間，並將結果保存在|outMediaUs|中
status_t getMediaTime( int64_t realUs, int64_t *outMediaUs,
bool allowPastMaxTime = false) const;
// query real time corresponding to media time 查詢與多媒體時間|targetMediaUs|對應的實際時間，結果保存在|outRealUs|中
status_t getRealTimeFor(int64_t targetMediaUs, int64_t *outRealUs) const;
private:
int64_t mAnchorTimeMediaUs;
int64_t mAnchorTimeRealUs;
int64_t mMaxTimeMediaUs;
int64_t mStartingTimeMediaUs;
float mPlaybackRate;
};
我把這個類的實現分為兩部分，不需要邏輯判斷的賦值或返回代碼，需要額外計算的代碼。先看簡單的部分，函數功能主要是賦值和返回參數。
// code from ~/frameworks/av/media/libstagefright/MediaClock.cpp
MediaClock::MediaClock() : mAnchorTimeMediaUs(-1), mAnchorTimeRealUs(-1),
mMaxTimeMediaUs(INT64_MAX), mStartingTimeMediaUs(-1), mPlaybackRate(1.0) {}
MediaClock::~MediaClock() {}
void MediaClock::setStartingTimeMedia(int64_t startingTimeMediaUs) {
mStartingTimeMediaUs = startingTimeMediaUs;
}
void MediaClock::clearAnchor() {
mAnchorTimeMediaUs = -1;
mAnchorTimeRealUs = -1;
}
void MediaClock::updateMaxTimeMedia(int64_t maxTimeMediaUs) {
mMaxTimeMediaUs = maxTimeMediaUs;
}
float MediaClock::getPlaybackRate() const {
Mutex::Autolock autoLock(mLock);
return mPlaybackRate;
}
這部分代碼實現了時鍾的主要功能，對多媒體時間和實際時間做了對應關系。（注意代碼部分有刪減，僅保留核心邏輯）
void MediaClock::updateAnchor(
int64_t anchorTimeMediaUs, // 錨點的播放時間戳
int64_t anchorTimeRealUs, // 錨點的實際時間
int64_t maxTimeMediaUs) {
int64_t nowUs = ALooper::GetNowUs(); // 當前系統時鍾
int64_t nowMediaUs = anchorTimeMediaUs + (nowUs - anchorTimeRealUs) * (double)mPlaybackRate; // 轉換為當前值，誤差低
if (maxTimeMediaUs != -1) {
mMaxTimeMediaUs = maxTimeMediaUs;
}
mAnchorTimeRealUs = nowUs;
mAnchorTimeMediaUs = nowMediaUs;
}
void MediaClock::setPlaybackRate(float rate) {
CHECK_GE(rate, 0.0);
if (mAnchorTimeRealUs == -1) {
mPlaybackRate = rate;
return;
}
int64_t nowUs = ALooper::GetNowUs();
mAnchorTimeMediaUs += (nowUs - mAnchorTimeRealUs) * (double)mPlaybackRate;
mAnchorTimeRealUs = nowUs;
mPlaybackRate = rate;
}
// 以下兩個函數完成MediaTime <-->realTime的映射，具體原理還是來自updateAnchor
status_t MediaClock::getMediaTime(int64_t realUs, int64_t *outMediaUs, bool allowPastMaxTime) const {
return getMediaTime_l(realUs, outMediaUs, allowPastMaxTime);
}
status_t MediaClock::getMediaTime_l(int64_t realUs, int64_t *outMediaUs, bool allowPastMaxTime) const {
if (mAnchorTimeRealUs == -1) {
return NO_INIT;
}
int64_t mediaUs = mAnchorTimeMediaUs
+ (realUs - mAnchorTimeRealUs) * (double)mPlaybackRate;
if (mediaUs > mMaxTimeMediaUs && !allowPastMaxTime) {
mediaUs = mMaxTimeMediaUs;
}
if (mediaUs < mStartingTimeMediaUs) {
mediaUs = mStartingTimeMediaUs;
}
if (mediaUs < 0) {
mediaUs = 0;
}
*outMediaUs = mediaUs;
return OK;
}
status_t MediaClock::getRealTimeFor(int64_t targetMediaUs, int64_t *outRealUs) const {
if (outRealUs == NULL) {
return BAD_VALUE;
}
if (mPlaybackRate == 0.0) {
return NO_INIT;
}
int64_t nowUs = ALooper::GetNowUs();
int64_t nowMediaUs;
status_t status =
getMediaTime_l(nowUs, &nowMediaUs, true /* allowPastMaxTime */);
if (status != OK) {
return status;
}
*outRealUs = (targetMediaUs - nowMediaUs) / (double)mPlaybackRate + nowUs;
return OK;
}
還記得在前面解釋Renderer::pause實現的時候把mPlaybackRate設置成0嘛，看到上面的計算代碼基本上就可以明白了。
比較有意思的是針對mPlaybackRate的處理及Renderer調用的邏輯。下面是獲得當前播放位置的函數實現
status_t NuPlayer::Renderer::getCurrentPosition(int64_t *mediaUs) {
// 注意是直接調用的MediaClock::getMediaTime()
status_t result = mMediaClock->getMediaTime(ALooper::GetNowUs(), mediaUs);
if (result == OK) {
return result;
}
// MediaClock未初始化，嘗試初始化之
{
AudioTimestamp ts;// 另一種時鍾計算方法
status_t res = mAudioSink->getTimestamp(ts);
if (res != OK) {
return result;
}
// AudioSink has rendered some frames.
int64_t nowUs = ALooper::GetNowUs();
int64_t nowMediaUs = mAudioSink->getPlayedOutDurationUs(nowUs)
+ mAudioFirstAnchorTimeMediaUs;
mMediaClock->updateAnchor(nowMediaUs, nowUs, -1);
}
return mMediaClock->getMediaTime(ALooper::GetNowUs(), mediaUs);
}
到這里基本解釋清楚MediaClock是做什么的，但是疑問還在，音視頻同步在哪里，怎么做到的？
5 AudioSink簡介
以下資料來在Google group，內容如下：
AudioTrack is the hardware audio sink. AudioSink is used for in-memory
decode and potentially other applications where output doesn't go
straight to hardware.
翻譯過來就是AudioTrack是一種特殊的AudioSink，與硬件對應；而AudioSink是用於內存解碼的，所得數據不直接輸出到音頻設備上。
在之前文章MediaPlayer Interface&State中可以看到MediaPlayerBase里面有一個抽象類定義，AudioSink。下面是具體的接口：
class AudioSink : public RefBase {
public:
enum cb_event_t {
CB_EVENT_FILL_BUFFER, // Request to write more data to buffer.
CB_EVENT_STREAM_END, // Sent after all the buffers queued in AF and HW are played
// back (after stop is called)
CB_EVENT_TEAR_DOWN // The AudioTrack was invalidated due to use case change:
// Need to re-evaluate offloading options
};
// Callback returns the number of bytes actually written to the buffer.
typedef size_t (*AudioCallback)(
AudioSink *audioSink, void *buffer, size_t size, void *cookie, cb_event_t event);
virtual ~AudioSink() {}
virtual bool ready() const = 0; // audio output is open and ready
virtual ssize_t bufferSize() const = 0;
virtual ssize_t frameCount() const = 0;
virtual ssize_t channelCount() const = 0;
virtual ssize_t frameSize() const = 0;
virtual uint32_t latency() const = 0;
virtual float msecsPerFrame() const = 0;
virtual status_t getPosition(uint32_t *position) const = 0;
virtual status_t getTimestamp(AudioTimestamp &ts) const = 0;
virtual int64_t getPlayedOutDurationUs(int64_t nowUs) const = 0;
virtual status_t getFramesWritten(uint32_t *frameswritten) const = 0;
virtual audio_session_t getSessionId() const = 0;
virtual audio_stream_type_t getAudioStreamType() const = 0;
virtual uint32_t getSampleRate() const = 0;
virtual int64_t getBufferDurationInUs() const = 0;
// If no callback is specified, use the "write" API below to submit audio data.
virtual status_t open(
uint32_t sampleRate, int channelCount, audio_channel_mask_t channelMask,
audio_format_t format=AUDIO_FORMAT_PCM_16_BIT,
int bufferCount=DEFAULT_AUDIOSINK_BUFFERCOUNT,
AudioCallback cb = NULL,
void *cookie = NULL,
audio_output_flags_t flags = AUDIO_OUTPUT_FLAG_NONE,
const audio_offload_info_t *offloadInfo = NULL,
bool doNotReconnect = false,
uint32_t suggestedFrameCount = 0) = 0;
virtual status_t start() = 0;
/* Input parameter |size| is in byte units stored in |buffer|.
* Data is copied over and actual number of bytes written (>= 0)
* is returned, or no data is copied and a negative status code
* is returned (even when |blocking| is true).
* When |blocking| is false, AudioSink will immediately return after
* part of or full |buffer| is copied over.
* When |blocking| is true, AudioSink will wait to copy the entire
* buffer, unless an error occurs or the copy operation is
* prematurely stopped.
*/
virtual ssize_t write(const void* buffer, size_t size, bool blocking = true) = 0;
virtual void stop() = 0;
virtual void flush() = 0;
virtual void pause() = 0;
virtual void close() = 0;
virtual status_t setPlaybackRate(const AudioPlaybackRate& rate) = 0;
virtual status_t getPlaybackRate(AudioPlaybackRate* rate /* nonnull */) = 0;
virtual bool needsTrailingPadding() { return true; }
virtual status_t setParameters(const String8& /* keyValuePairs */) { return NO_ERROR; }
virtual String8 getParameters(const String8& /* keys */) { return String8::empty(); }
};
在Renderer的構造函數中可以看到AudioSink是由NuPlayer傳遞過來的。明顯的這僅僅是通過抽象實現了在Renderer中操作AudioSink及其子類的邏輯。當然在實際使用中，AudioSink也可以作為播放時間的參考，比如上面的getCurrentPosition的實現。這里面的open/close/start/stop/flush/pause/write接口均在Renderer中調用過，后續針對同步的解釋會詳細說明的。
6 VideoFrameScheduler簡介
看名字，感覺這個功能跟MediaClock類似，只是專門針對視頻幀的處理邏輯，這也是libstagefright提供的一個公共類，實際上是做視頻渲染調整的，以保證視頻渲染時間在VSYNC時間之后，防止出現畫面撕裂的情況。其對外接口如下：
struct VideoFrameScheduler : public RefBase {
VideoFrameScheduler();
// (re)initialize scheduler 初始化，給定幀率
void init(float videoFps = -1);
// 僅在視頻渲染時間不連續的情況下使用，比如seek
void restart();
// 通過renderTime計算視頻幀的調整時間（單位納秒）
nsecs_t schedule(nsecs_t renderTime);
// 返回主屏的垂直同步間隔
nsecs_t getVsyncPeriod();
// 返回幀率
float getFrameRate();
void release();
}
內部實現我就不做解釋了，基本意思還是從Renderer的調用中說起。Renderer中主要調用了VideoFrameScheduler的以下接口：
mVideoScheduler = new VideoFrameScheduler();
mVideoScheduler->init(fps);
mVideoScheduler->restart(); // 以下調用都在postDrainVideoQueue中
realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;
int64_t twoVsyncsUs = 2 * (mVideoScheduler->getVsyncPeriod() / 1000);
7 音視頻同步時如何實現的？
從Renderer接口層來看，沒有任何關於同步處理的接口，僅有有限的幾個控制接口flush/pause/resume，以及queueBuffer/queueEOS接口。同步問題的核心就在於ALooper-AHandler機制。其實真正的同步都是在消息循環的響應函數里實現的。先看音頻。
Renderer中的音頻同步機制
起始位置從音頻PCM數據進入開始，處理在Renderer::queueBuffer()中，最終發送了kWhatQueueBuffer消息。這個消息的實際處理函數是Renderer::onQueueBuffer()。實際代碼在“音視頻原始數據輸入——queueBuffer”中有，這里僅針對音頻流程解釋下。 基本邏輯很簡單，保存傳入的buffer參數，並通知輸出下AudioQueue。
QueueEntry entry;
Mutex::Autolock autoLock(mLock);
mAudioQueue.push_back(entry);
postDrainAudioQueue_l();
下面看看postDrainAudioQueue_l的實現，內部實現邏輯基本上就是邊界判斷加上發送kWhatDrainAudioQueue消息。
void NuPlayer::Renderer::postDrainAudioQueue_l(int64_t delayUs) {
if (mAudioQueue.empty()) return;
mDrainAudioQueuePending = true;
sp<AMessage> msg = new AMessage(kWhatDrainAudioQueue, this);
msg->setInt32("drainGeneration", mAudioDrainGeneration);
msg->post(delayUs);
}
那就繼續查看下這個消息如何處理的。
case kWhatDrainAudioQueue:
{
mDrainAudioQueuePending = false;
if (onDrainAudioQueue()) {
uint32_t numFramesPlayed;
uint32_t numFramesPendingPlayout = mNumFramesWritten - numFramesPlayed;
// 這里是audio sink中緩存了多長的可用於播放的數據
int64_t delayUs = mAudioSink->msecsPerFrame() * numFramesPendingPlayout * 1000ll;
if (mPlaybackRate > 1.0f) {
delayUs /= mPlaybackRate;
}
// 利用一半的延時來保證下次刷新時間（注意時間上有重疊）
delayUs /= 2;
// 參考buffer大小來估計最大的延時時間
const int64_t maxDrainDelayUs = std::max(
mAudioSink->getBufferDurationInUs(), (int64_t)500000 /* half second */);
ALOGD_IF(delayUs > maxDrainDelayUs, "postDrainAudioQueue long delay: %lld > %lld",
(long long)delayUs, (long long)maxDrainDelayUs);
Mutex::Autolock autoLock(mLock);
postDrainAudioQueue_l(delayUs); // 這里同一個消息重發了
}
break;
}
到這里，貌似還是沒有同步的機制，不過我們已經知道這個音頻播放消息的觸發機制了，在queueBuffer和消息處理函數中都會觸發，基本上就是定時器。還有最后一個函數onDrainAudioQueue()。下面是代碼：
bool NuPlayer::Renderer::onDrainAudioQueue() {
uint32_t numFramesPlayed;
if (mAudioSink->getPosition(&numFramesPlayed) != OK) {
drainAudioQueueUntilLastEOS();
ALOGW("onDrainAudioQueue(): audio sink is not ready");
return false;
}
uint32_t prevFramesWritten = mNumFramesWritten;
while (!mAudioQueue.empty()) {
QueueEntry *entry = &*mAudioQueue.begin();
mLastAudioBufferDrained = entry->mBufferOrdinal;
if (entry->mBuffer == NULL) {
// 刪除針對EOS的處理代碼
}
// ignore 0-sized buffer which could be EOS marker with no data
if (entry->mOffset == 0 && entry->mBuffer->size() > 0) {
int64_t mediaTimeUs;
CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));
ALOGV("onDrainAudioQueue: rendering audio at media time %.2f secs",
mediaTimeUs / 1E6);
onNewAudioMediaTime(mediaTimeUs);
}
size_t copy = entry->mBuffer->size() - entry->mOffset;
ssize_t written = mAudioSink->write(entry->mBuffer->data() + entry->mOffset,
copy, false /* blocking */);
if (written < 0) {/* ...忽略異常處理部分代碼 */}
entry->mOffset += written;
size_t remainder = entry->mBuffer->size() - entry->mOffset;
if ((ssize_t)remainder < mAudioSink->frameSize()) {
if (remainder > 0) {// 這是直接湊成完整的一幀音頻
ALOGW("Corrupted audio buffer has fractional frames, discarding %zu bytes.", remainder);
entry->mOffset += remainder;
copy -= remainder;
}
entry->mNotifyConsumed->post();
mAudioQueue.erase(mAudioQueue.begin());
entry = NULL;
}
size_t copiedFrames = written / mAudioSink->frameSize();
mNumFramesWritten += copiedFrames;
{
Mutex::Autolock autoLock(mLock);
int64_t maxTimeMedia;
maxTimeMedia = mAnchorTimeMediaUs +
(int64_t)(max((long long)mNumFramesWritten - mAnchorNumFramesWritten, 0LL)
* 1000LL * mAudioSink->msecsPerFrame());
mMediaClock->updateMaxTimeMedia(maxTimeMedia);
notifyIfMediaRenderingStarted_l();
}
if (written != (ssize_t)copy) {
// A short count was received from AudioSink::write()
//
// AudioSink write is called in non-blocking mode.
// It may return with a short count when:
//
// 1) Size to be copied is not a multiple of the frame size. Fractional frames are
// discarded.
// 2) The data to be copied exceeds the available buffer in AudioSink.
// 3) An error occurs and data has been partially copied to the buffer in AudioSink.
// 4) AudioSink is an AudioCache for data retrieval, and the AudioCache is exceeded.
// (Case 1)
// Must be a multiple of the frame size. If it is not a multiple of a frame size, it
// needs to fail, as we should not carry over fractional frames between calls.
CHECK_EQ(copy % mAudioSink->frameSize(), 0);
// (Case 2, 3, 4)
// Return early to the caller.
// Beware of calling immediately again as this may busy-loop if you are not careful.
ALOGV("AudioSink write short frame count %zd < %zu", written, copy);
break;
}
}
// calculate whether we need to reschedule another write.
bool reschedule = !mAudioQueue.empty()
&& (!mPaused
|| prevFramesWritten != mNumFramesWritten); // permit pause to fill buffers
//ALOGD("reschedule:%d empty:%d mPaused:%d prevFramesWritten:%u mNumFramesWritten:%u",
// reschedule, mAudioQueue.empty(), mPaused, prevFramesWritten, mNumFramesWritten);
return reschedule;
}
這里面比較主要的更新是onNewAudioMediaTime和mNumFramesWritten字段。
剩下的一部分代碼是關於異常邊界情況下的音視頻處理邏輯：
sp<ABuffer> firstAudioBuffer = (*mAudioQueue.begin()).mBuffer;
sp<ABuffer> firstVideoBuffer = (*mVideoQueue.begin()).mBuffer;
if (firstAudioBuffer == NULL || firstVideoBuffer == NULL) {
// 對於一個隊列為空的情況，通知另個一隊列EOS
syncQueuesDone_l();
return;
}
int64_t firstAudioTimeUs;
int64_t firstVideoTimeUs;
CHECK(firstAudioBuffer->meta()
->findInt64("timeUs", &firstAudioTimeUs));
CHECK(firstVideoBuffer->meta()
->findInt64("timeUs", &firstVideoTimeUs));
int64_t diff = firstVideoTimeUs - firstAudioTimeUs;
if (diff > 100000ll) {
// 音頻數據時間戳比視頻數據早0.1s，
(*mAudioQueue.begin()).mNotifyConsumed->post();
mAudioQueue.erase(mAudioQueue.begin());
return;
}
syncQueuesDone_l();
Renderer中的視頻同步部分
和音頻同步類似，入口在在Renderer::queueBuffer()，主要區分在Renderer::onQueueBuffer()中，代碼如下：
// 如果是視頻，則將數據存放到視頻隊列，然后安排刷新
mVideoQueue.push_back(entry);
postDrainVideoQueue();
下面按照之前的思路繼續分析，接下來是postDrainVideoQueue實現，主要音視頻同步邏輯位於這里。
void NuPlayer::Renderer::postDrainVideoQueue() {
if (mVideoQueue.empty()) {
return;
}
QueueEntry &entry = *mVideoQueue.begin();
sp<AMessage> msg = new AMessage(kWhatDrainVideoQueue, this); //這是實際處理視頻緩沖區和顯示的消息
msg->setInt32("drainGeneration", getDrainGeneration(false /* audio */));
if (entry.mBuffer == NULL) {
// EOS doesn't carry a timestamp.
msg->post();
mDrainVideoQueuePending = true;
return;
}
bool needRepostDrainVideoQueue = false;
int64_t delayUs;
int64_t nowUs = ALooper::GetNowUs();
int64_t realTimeUs;
int64_t mediaTimeUs;
CHECK(entry.mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));
if (mFlags & FLAG_REAL_TIME) {
realTimeUs = mediaTimeUs;
} else {
{
Mutex::Autolock autoLock(mLock);
if (mAnchorTimeMediaUs < 0) { // 同步基准未設置的情況下，直接顯示
mMediaClock->updateAnchor(mediaTimeUs, nowUs, mediaTimeUs);
mAnchorTimeMediaUs = mediaTimeUs;
realTimeUs = nowUs;
} else if (!mVideoSampleReceived) { // 第一幀未顯示前，直接顯示
// Always render the first video frame.
realTimeUs = nowUs;
} else if (mAudioFirstAnchorTimeMediaUs < 0 // 音頻未播放之前，以視頻為准
|| mMediaClock->getRealTimeFor(mediaTimeUs, &realTimeUs) == OK) {
realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);
} else if (mediaTimeUs - mAudioFirstAnchorTimeMediaUs >= 0) { // 視頻超前的情況下，等待
needRepostDrainVideoQueue = true;
realTimeUs = nowUs;
} else {
realTimeUs = nowUs;
}
}
// Heuristics to handle situation when media time changed without a
// discontinuity. If we have not drained an audio buffer that was
// received after this buffer, repost in 10 msec. Otherwise repost
// in 500 msec.
delayUs = realTimeUs - nowUs;
int64_t postDelayUs = -1;
if (delayUs > 500000) {
postDelayUs = 500000;
if (mHasAudio && (mLastAudioBufferDrained - entry.mBufferOrdinal) <= 0) {
postDelayUs = 10000;
}
} else if (needRepostDrainVideoQueue) {
// CHECK(mPlaybackRate > 0);
// CHECK(mAudioFirstAnchorTimeMediaUs >= 0);
// CHECK(mediaTimeUs - mAudioFirstAnchorTimeMediaUs >= 0);
postDelayUs = mediaTimeUs - mAudioFirstAnchorTimeMediaUs;
postDelayUs /= mPlaybackRate;
}
if (postDelayUs >= 0) {
msg->setWhat(kWhatPostDrainVideoQueue);
msg->post(postDelayUs);
mVideoScheduler->restart();
ALOGI("possible video time jump of %dms or uninitialized media clock, retrying in %dms",
(int)(delayUs / 1000), (int)(postDelayUs / 1000));
mDrainVideoQueuePending = true;
return;
}
}
realTimeUs = mVideoScheduler->schedule(realTimeUs * 1000) / 1000;
int64_t twoVsyncsUs = 2 * (mVideoScheduler->getVsyncPeriod() / 1000);
delayUs = realTimeUs - nowUs;
// 上面代碼的主要目的是計算這個延時
ALOGW_IF(delayUs > 500000, "unusually high delayUs: %" PRId64, delayUs);
// post 2 display refreshes before rendering is due
msg->post(delayUs > twoVsyncsUs ? delayUs - twoVsyncsUs : 0);
mDrainVideoQueuePending = true;
}
這里主要的是發送了一個延時消息kWhatDrainVideoQueue，下面是如何處理的代碼：
case kWhatDrainVideoQueue:
{
int32_t generation;
CHECK(msg->findInt32("drainGeneration", &generation));
if (generation != getDrainGeneration(false /* audio */)) {
break;
}
mDrainVideoQueuePending = false;
onDrainVideoQueue();
postDrainVideoQueue(); // 注意這里相當於定時器的實現了
break;
}
直接調用onDrainVideoQueue函數，看看如何實現的：
void NuPlayer::Renderer::onDrainVideoQueue() {
if (mVideoQueue.empty()) {
return;
}
QueueEntry *entry = &*mVideoQueue.begin();
if (entry->mBuffer == NULL) {
// ...省略針對EOS 處理
}
int64_t nowUs = ALooper::GetNowUs();
int64_t realTimeUs;
int64_t mediaTimeUs = -1;
if (mFlags & FLAG_REAL_TIME) {
CHECK(entry->mBuffer->meta()->findInt64("timeUs", &realTimeUs));
} else {
CHECK(entry->mBuffer->meta()->findInt64("timeUs", &mediaTimeUs));
realTimeUs = getRealTimeUs(mediaTimeUs, nowUs);
}
bool tooLate = false;
if (!mPaused) {
setVideoLateByUs(nowUs - realTimeUs);
tooLate = (mVideoLateByUs > 40000);
if (tooLate) {
ALOGV("video late by %lld us (%.2f secs)",
(long long)mVideoLateByUs, mVideoLateByUs / 1E6);
} else {
int64_t mediaUs = 0;
mMediaClock->getMediaTime(realTimeUs, &mediaUs);
ALOGV("rendering video at media time %.2f secs",
(mFlags & FLAG_REAL_TIME ? realTimeUs :
mediaUs) / 1E6);
if (!(mFlags & FLAG_REAL_TIME)
&& mLastAudioMediaTimeUs != -1
&& mediaTimeUs > mLastAudioMediaTimeUs) {
// If audio ends before video, video continues to drive media clock.
// Also smooth out videos >= 10fps.
mMediaClock->updateMaxTimeMedia(mediaTimeUs + 100000);
}
}
} else {
setVideoLateByUs(0);
if (!mVideoSampleReceived && !mHasAudio) {
// This will ensure that the first frame after a flush won't be used as anchor
// when renderer is in paused state, because resume can happen any time after seek.
Mutex::Autolock autoLock(mLock);
clearAnchorTime_l();
}
}
// Always render the first video frame while keeping stats on A/V sync.
if (!mVideoSampleReceived) {
realTimeUs = nowUs;
tooLate = false;
}
entry->mNotifyConsumed->setInt64("timestampNs", realTimeUs * 1000ll); // 上面所有計算的參數在這里使用了
entry->mNotifyConsumed->setInt32("render", !tooLate);
entry->mNotifyConsumed->post(); // 注意這里，實際是向解碼器發送消息，用於顯示
mVideoQueue.erase(mVideoQueue.begin());
entry = NULL;
mVideoSampleReceived = true;
if (!mPaused) { // 這里是通知NuPlayer層渲染開始
if (!mVideoRenderingStarted) {
mVideoRenderingStarted = true;
notifyVideoRenderingStart();
}
Mutex::Autolock autoLock(mLock);
notifyIfMediaRenderingStarted_l();
}
}
到這里，小結下，讀完這部分代碼發現，NuPlayer::Renderer使用的以視頻為基准的同步機制，音頻晚了直接丟包，視頻需要顯示。同步主要位於視頻緩沖區處理部分onDrainVideoQueue和音頻緩沖區處理部分onDrainVideoQueue中。音視頻的渲染都是采用類似定時器的機制，只不過視頻顯示需要依賴於實際解碼器，音頻播放需要依賴於AudioSink的接口。
8 總結
本文主要參考NuPlayer::Renderer的代碼做的分析，持續時間比較長。我都懷疑自己具體寫的對不對。
非常抱歉拖了這么久，文中代碼比較多，如果諸位絕對不對胃口可以略過。
怎么說呢？ Renderer涉及部分比較多，包括NuPlayer、AudioSink、MediaClock、VideoScheduler等。細節還是有待分析，不過基本整理情況是什么了。
我到現在才認識到理解和整理出來的差距。還需要多歷練下。