ffmpeg + rtp介紹

1）ffmpeg如何判斷一幀數據是正確的？ffmpeg有沒有錯誤處理的模式，能使花屏的幀(h264格式的)不顯示出來？

2) H264網絡傳輸過程中丟包造成馬賽克問題?

 

原因：

1. 接收網絡數據包后沒有調整包的順序，譬如說接受包的順序是1，3，4，2，如果沒有調整順序的話，發送給解碼器的順序也是1，3，4，2，這樣肯定會出現馬賽克 ； 2. 接收網絡數據包后沒有沒有合並數據包，眾所周知，一個Video幀可能被分割成多個網絡數據包傳送，譬如說是1，2，3，如果在接受端沒有將這三個包合並成一個Video幀發送給解碼器，而是當成三個Video幀發送給解碼器，也肯定會出現馬賽克 ； 3. 沒有正確處理好網絡丟包的情況，如果所丟的數據包正好傳送的是另外一個幀所參考的的數據，這樣也會出現馬賽克 ； 4. 解碼器有問題，如果播放本地文件也出現馬賽克的話。

 

解決方法：

1.服務器軟件用多線程： （1）主線程：讀出（看你的圖象具體怎么上PC機了）一幀視頻數據，送給拆分線程。
（2）拆分線程：接到一幀視頻，開始拆包、做幀標記、打序列號，送給發送線程。 （3）發送線程：用RTP socket把封裝好的數據包發給客戶端。此socket是點對多點、單向 有根方式的組播套接字，實際上是基於UDP派生的，但他用到了RTP和RTCP（實時傳輸 協議和實時傳輸控制協議），如果你傳輸的不是實時數據，直接用UDP就行了。
2.客戶端軟件結構一般用多線程，線程間用事件對象進行同步，而共享數據區用臨界區對象進
行同步。 （1）主線程：接收網絡數據包的線程，優先級最高，可以盡量保證不丟數據，也采用RTP協 議,用網絡事件來觸發。主線程接收到視頻數據包后，將數據放入一個鏈表中，然后用事件對象觸發組裝線程。 （2）組裝線程：從鏈表中讀出數據包，要進行幀確認、排序等工作，當把一幀圖象的所有 包都取到時，再調用組裝模塊（可以是一個函數），將這些數據包組裝成完整的一個幀，然后送到解壓線程。 （3）若干解壓播放線程。主要考慮到如果你客戶端軟件想同時播放多畫面，比如說4畫面圖 象，就要用4個解壓播放線程。 （4）至於圖象存儲，要看你的客戶需要怎么存了，如果是手工存當然不需要單開線程，如果 是規定定時存或在某個事件發生時自動存盤，就需要單開一個線程，由定時器到時消息或此事件發生來觸發。
后來我們項目也將圖象送上了廣域網和Internet，主要就是傳輸的速率明顯慢了。把服務器軟件放在Web服務器上，用UDP點對點的方式給提出視頻服務的Internet客戶端發出相應視頻幀。還可以實現對每個客戶端的各種服務請求的響應和控制。

建議：

（1）Winsock傳輸部分最好不要用MFC提供的類，自己用API做，但是應用 程序可以用MFC做，
（2）在局域網上用點對多點的組播方式傳輸時，基於RTP和RTCP協議來做。 服務器方得拆包傳，每包最好不要大於2K，以我去年做的項目經驗， 用1.5K比較好，每包間隔1-2毫秒。發送方給每包打上時間戳和序列 號，接收方重新排序和組裝。 （3）如果是點對點傳，應該基於UDP協議，可以一幀一幀的傳，我最大傳過 30K。沒問題，我已經在實際項目中試過，JPEG格式的視頻流在局域網 中1秒15幀，一幀12K；在廣域網中1秒中4幀，一幀15K。 （4） 如果你傳輸的是監控的告警數據，要求准確性，你必須基於TCP協議，用點對點方式傳輸。

 

VLC 對應的解決策略：

static inline uint16_t rtp_seq (const block_t *block)
{
    assert (block->i_buffer >= 4);
    return GetWBE (block->p_buffer + 2);
}

#define GetWBE( p )     U16_AT( p )

/* MSB (big endian)/LSB (little endian) conversions - network order is always
 * MSB, and should be used for both network communications and files. */
LIBVLC_USED
static inline uint16_t U16_AT( const void * _p )
{
    const uint8_t * p = (const uint8_t *)_p;
    return ( ((uint16_t)p[0] << 8) | p[1] );
}

/** State for a RTP session: */
struct rtp_session_t
{
    rtp_source_t **srcv;
    unsigned       srcc;
    uint8_t        ptc;
    rtp_pt_t      *ptv;
};

/** State for an RTP source */
struct rtp_source_t
{
    uint32_t ssrc;
    uint32_t jitter;  /* interarrival delay jitter estimate */
    mtime_t  last_rx; /* last received packet local timestamp */
    uint32_t last_ts; /* last received packet RTP timestamp */

    uint32_t ref_rtp; /* sender RTP timestamp reference */
    mtime_t  ref_ntp; /* sender NTP timestamp reference */

    uint16_t bad_seq; /* tentatively next expected sequence for resync */
    uint16_t max_seq; /* next expected sequence */

    uint16_t last_seq; /* sequence of the next dequeued packet */
    block_t *blocks; /* re-ordered blocks queue */
    void    *opaque[0]; /* Per-source private payload data */
};

/**
 * Destroys an RTP source and its associated streams.
 */
static void
rtp_source_destroy (demux_t *demux, const rtp_session_t *session,
                    rtp_source_t *source)
{
    msg_Dbg (demux, "removing RTP source (%08x)", source->ssrc);

    for (unsigned i = 0; i < session->ptc; i++)
        session->ptv[i].destroy (demux, source->opaque[i]);
    block_ChainRelease (source->blocks);
    free (source);
}

/**
 * Receives an RTP packet and queues it. Not a cancellation point.
 *
 * @param demux VLC demux object
 * @param session RTP session receiving the packet
 * @param block RTP packet including the RTP header
 */
void
rtp_queue (demux_t *demux, rtp_session_t *session, block_t *block)
{
    demux_sys_t *p_sys = demux->p_sys;

    /* RTP header sanity checks (see RFC 3550) */
    if (block->i_buffer < 12)   //如果RTP包的長度小於12，說明包傳輸有錯誤
        goto drop;
    if ((block->p_buffer[0] >> 6 ) != 2) /* RTP version number(rtp版本號必須為2) */
        goto drop;

    /* Remove padding if present (判斷RTP數據包是否有填充字節，如果有填充字節，解析數據包時，必須將填充字節去掉)*/
    if (block->p_buffer[0] & 0x20)
    {
        uint8_t padding = block->p_buffer[block->i_buffer - 1];
        if ((padding == 0) || (block->i_buffer < (12u + padding)))
            goto drop; /* illegal value */

        block->i_buffer -= padding;
    }

    mtime_t        now = mdate ();   //獲取精確的時鍾信息
    rtp_source_t  *src  = NULL;
    const uint16_t seq  = rtp_seq (block); //獲取序列號
    const uint32_t ssrc = GetDWBE (block->p_buffer + 8); //獲取SSRC

    /* In most case, we know this source already 找到相同的SSRC*/
    for (unsigned i = 0, max = session->srcc; i < max; i++)
    {
        rtp_source_t *tmp = session->srcv[i];
        if (tmp->ssrc == ssrc)
        {
            src = tmp;
            break;
        }

        /* RTP source garbage collection */
        if ((tmp->last_rx + p_sys->timeout) < now)    //超時了
        {
            rtp_source_destroy (demux, session, tmp);
            if (--session->srcc > 0)
                session->srcv[i] = session->srcv[session->srcc - 1]; //將最后一個賦值給刪除的那個ssrc
        }
    }

    if (src == NULL)  //在原來的會話中沒有找到同樣的ssrc，說明是新來的ssrc
    {
        /* New source */
        if (session->srcc >= p_sys->max_src)  //判斷是不是到達了最大的ssrc數的極限值(max)
        {
            msg_Warn (demux, "too many RTP sessions");
            goto drop;
        }

        rtp_source_t **tab;
        tab = realloc (session->srcv, (session->srcc + 1) * sizeof (*tab));
        if (tab == NULL)
            goto drop;
        session->srcv = tab;

        src = rtp_source_create (demux, session, ssrc, seq); //創建ssrc
        if (src == NULL)
            goto drop;

        tab[session->srcc++] = src;
        /* Cannot compute jitter yet */
    }
    else
    {
        const rtp_pt_t *pt = rtp_find_ptype (session, src, block, NULL);

        if (pt != NULL)
        {
            /* Recompute jitter estimate.
             * That is computed from the RTP timestamps and the system clock.
             * It is independent of RTP sequence. */
            uint32_t freq = pt->frequency;
            int64_t ts = rtp_timestamp (block);
            int64_t d = ((now - src->last_rx) * freq) / CLOCK_FREQ;
            d        -=    ts - src->last_ts;
            if (d < 0) d = -d;
            src->jitter += ((d - src->jitter) + 8) >> 4;
        }
    }
    src->last_rx = now;
    block->i_pts = now; /* store reception time until dequeued */
    src->last_ts = rtp_timestamp (block);

    /* Check sequence number */
    /* NOTE: the sequence number is per-source,
     * but is independent from the payload type. */
    int16_t delta_seq = seq - src->max_seq;
    if ((delta_seq > 0) ? (delta_seq > p_sys->max_dropout)
                        : (-delta_seq > p_sys->max_misorder))
    {
        msg_Dbg (demux, "sequence discontinuity"
                 " (got: %"PRIu16", expected: %"PRIu16")", seq, src->max_seq);
        if (seq == src->bad_seq)
        {
            src->max_seq = src->bad_seq = seq + 1;
            src->last_seq = seq - 0x7fffe; /* hack for rtp_decode() */
            msg_Warn (demux, "sequence resynchronized");
            block_ChainRelease (src->blocks);
            src->blocks = NULL;
        }
        else
        {
            src->bad_seq = seq + 1;
            goto drop;
        }
    }
    else
    if (delta_seq >= 0)
        src->max_seq = seq + 1;

    /* Queues the block in sequence order,
     * hence there is a single queue for all payload types. */
    block_t **pp = &src->blocks;
    for (block_t *prev = *pp; prev != NULL; prev = *pp)  //將接收到的數據插入到隊列中
    {
        int16_t delta_seq = seq - rtp_seq (prev);
        if (delta_seq < 0)
            break;
        if (delta_seq == 0)
        {
            msg_Dbg (demux, "duplicate packet (sequence: %"PRIu16")", seq);
            goto drop; /* duplicate */
        }
        pp = &prev->p_next;
    }
    block->p_next = *pp;
    *pp = block;

    /*rtp_decode (demux, session, src);*/
    return;

drop:
    block_Release (block);
}


 

 vlc調用過程：

/**
 * @file input.c
 * @brief RTP packet input
 */
/*****************************************************************************
 * Copyright 漏 2008 R茅mi Denis-Courmont
 *
 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public License
 * as published by the Free Software Foundation; either version 2.1
 * of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 ****************************************************************************/

#ifdef HAVE_CONFIG_H
# include <config.h>
#endif

#include <vlc_common.h>
#include <vlc_demux.h>
#include <vlc_block.h>
#include <vlc_network.h>

#include <unistd.h>
#ifdef HAVE_POLL
# include <poll.h>
#endif

#include "rtp.h"
#ifdef HAVE_SRTP
# include <srtp.h>
#endif

static bool fd_dead (int fd)
{
    struct pollfd ufd = { .fd = fd, };
    return (poll (&ufd, 1, 0) > 0) && (ufd.revents & POLLHUP);
}

/**
 * Gets a datagram from the network.
 * @param fd datagram file descriptor
 * @return a block or NULL on fatal error (socket dead)
 */
static block_t *rtp_dgram_recv (vlc_object_t *obj, int fd)
{
    block_t *block = block_Alloc (0xffff);
    ssize_t len;

    block_cleanup_push (block);
    do
    {
        len = net_Read (obj, fd, NULL,
                        block->p_buffer, block->i_buffer, false);

        if (((len <= 0) && fd_dead (fd)) || !vlc_object_alive (obj))
        {   /* POLLHUP -> permanent (DCCP) socket error */
            block_Release (block);
            block = NULL;
            break;
        }
    }
    while (len == -1);
    vlc_cleanup_pop ();

    return block ? block_Realloc (block, 0, len) : NULL;
}


/**
 * Gets a framed RTP packet.
 * @param fd stream file descriptor
 * @return a block or NULL in case of fatal error
 */
static block_t *rtp_stream_recv (vlc_object_t *obj, int fd)
{
    ssize_t len = 0;
    uint8_t hdr[2]; /* frame header */

    /* Receives the RTP frame header */
    do
    {
        ssize_t val = net_Read (obj, fd, NULL, hdr + len, 2 - len, false);
        if (val <= 0)
            return NULL;
        len += val;
    }
    while (len < 2);

    block_t *block = block_Alloc (GetWBE (hdr));

    /* Receives the RTP packet */
    for (ssize_t i = 0; i < len;)
    {
        ssize_t val;

        block_cleanup_push (block);
        val = net_Read (obj, fd, NULL,
                        block->p_buffer + i, block->i_buffer - i, false);
        vlc_cleanup_pop ();

        if (val <= 0)
        {
            block_Release (block);
            return NULL;
        }
        i += val;
    }

    return block;
}


static block_t *rtp_recv (demux_t *demux)
{
    demux_sys_t *p_sys = demux->p_sys;

    for (block_t *block;; block_Release (block))
    {
        block = p_sys->framed_rtp
                ? rtp_stream_recv (VLC_OBJECT (demux), p_sys->fd)
                : rtp_dgram_recv (VLC_OBJECT (demux), p_sys->fd);
        if (block == NULL)
        {
            msg_Err (demux, "RTP flow stopped");
            break; /* fatal error */
        }

        if (block->i_buffer < 2)
            continue;

        /* FIXME */
        const uint8_t ptype = rtp_ptype (block);
        if (ptype >= 72 && ptype <= 76)
            continue; /* Muxed RTCP, ignore for now */
#ifdef HAVE_SRTP
        if (p_sys->srtp)
        {
            size_t len = block->i_buffer;
            int canc, err;

            canc = vlc_savecancel ();
            err = srtp_recv (p_sys->srtp, block->p_buffer, &len);
            vlc_restorecancel (canc);
            if (err)
            {
                msg_Dbg (demux, "SRTP authentication/decryption failed");
                continue;
            }
            block->i_buffer = len;
        }
#endif
        return block; /* success! */
    }
    return NULL;
}


static void timer_cleanup (void *timer)
{
    vlc_timer_destroy ((vlc_timer_t)timer);
}

static void rtp_process (void *data);

void *rtp_thread (void *data)
{
    demux_t *demux = data;
    demux_sys_t *p_sys = demux->p_sys;
    bool autodetect = true;

    if (vlc_timer_create (&p_sys->timer, rtp_process, data))
        return NULL;
    vlc_cleanup_push (timer_cleanup, (void *)p_sys->timer);

    for (;;)
    {
        block_t *block = rtp_recv (demux);
        if (block == NULL)
            break;

        if (autodetect)
        {   /* Autodetect payload type, _before_ rtp_queue() */
            /* No need for lock - the queue is empty. */
            if (rtp_autodetect (demux, p_sys->session, block))
            {
                block_Release (block);
                continue;
            }
            autodetect = false;
        }

        int canc = vlc_savecancel ();
        vlc_mutex_lock (&p_sys->lock);
        rtp_queue (demux, p_sys->session, block);
        vlc_mutex_unlock (&p_sys->lock);
        vlc_restorecancel (canc);

        rtp_process (demux);
    }
    vlc_cleanup_run ();
    return NULL;
}


/**
 * Process one RTP packet from the de-jitter queue.
 */
static void rtp_process (void *data)
{
    demux_t *demux = data;
    demux_sys_t *p_sys = demux->p_sys;
    mtime_t deadline;

    vlc_mutex_lock (&p_sys->lock);
    if (rtp_dequeue (demux, p_sys->session, &deadline))
        vlc_timer_schedule (p_sys->timer, true, deadline, 0);
    vlc_mutex_unlock (&p_sys->lock);
}