NDK学习笔记：FFmpeg音视频同步3

本篇内容说多不多，但如果要说得明明白白的，可能就有点难度了。所以我决定把我的调试过程日志都呈现出来，方便大家理解。继上一篇文末，我们学习到了什么是DTS/PTS，还有音视同步的三种策略。那再FFmpeg框架中，又怎么体现这些变量？我们以音频的解码线程为例，如下代码所示。

void* audio_avframe_decoder(void* arg)
{JNIEnv *env = NULL;if ( (*gJavaVM)->AttachCurrentThread(gJavaVM, &env,NULL) != JNI_OK) {LOGE("gJavaVM->Env Error!\n");pthread_exit((void *) -1);}SyncPlayer* player = (SyncPlayer*)arg;AVCodecContext* audioCodecCtx = player->input_codec_ctx[player->audio_stream_index];AV_PACKET_BUFFER* audioAVPacketButter = player->audio_avpacket_buffer;AVStream *audioStream = player->input_format_ctx->streams[player->audio_stream_index];int64_t audioDuration = (int64_t) (audioStream->duration * av_q2d(audioStream->time_base));LOGI("audio steam time_base : %d/%d \n",audioStream->time_base.num, audioStream->time_base.den);AVFrame *frame = av_frame_alloc();//16bit 44100 PCM 数据的实际内存空间。uint8_t *out_buffer = (uint8_t *)av_malloc(MAX_AUDIO_FRAME_SIZE);// AudioTrack.play(*env)->CallVoidMethod(env, player->audio_track, player->audio_track_play_mid);int ret;int64_t pts;while(player->stop_thread_audio_decoder == 0){pthread_mutex_lock(&audioAVPacketButter->mutex);AVPacket* packet = get_read_packet(audioAVPacketButter);pthread_mutex_unlock(&audioAVPacketButter->mutex);//AVPacket->AVFrameret = avcodec_send_packet(audioCodecCtx, packet);if (ret == AVERROR_EOF){av_packet_unref(packet);LOGW("audio_decoder avcodec_send_packet：%d\n", ret);break; // 跳出 while(player->stop_thread_audio_decoder==0)}else if(ret < 0){av_packet_unref(packet);LOGE("audio_decoder avcodec_send_packet：%d\n", ret);continue;}while(ret >= 0){ret = avcodec_receive_frame(audioCodecCtx, frame);if (ret == AVERROR(EAGAIN) ) {//LOGD("audio_decoder avcodec_receive_frame：%d\n", ret);break; // 跳出 while(ret>=0)} else if (ret < 0 || ret == AVERROR_EOF) {LOGW("audio_decoder avcodec_receive_frame：%d %s\n", AVERROR(ret), av_err2str(ret));av_packet_unref(packet);goto end;  //end处进行资源释放等善后处理}// !test startif ((pts = av_frame_get_best_effort_timestamp(frame)) == AV_NOPTS_VALUE)pts = 0;LOGI("audio current frame PTS : %lld\n",pts);pts *= av_q2d(audioStream->time_base);LOGI("audio current frame PTS : %lld\n",pts);// !test endif (ret >= 0){swr_convert(player->swr_ctx, &out_buffer, MAX_AUDIO_FRAME_SIZE, (const uint8_t **) frame->data, frame->nb_samples);//获取sample的sizeint out_buffer_size = av_samples_get_buffer_size(NULL, player->out_channel_nb,frame->nb_samples, player->out_sample_fmt, 1);//AudioTrack.write(byte[] int int) 需要byte数组,对应jni的jbyteArray//需要把out_buffer缓冲区数据转成byte数组jbyteArray audio_data_byteArray = (*env)->NewByteArray(env, out_buffer_size);jbyte* fp_AudioDataArray = (*env)->GetByteArrayElements(env, audio_data_byteArray, NULL);memcpy(fp_AudioDataArray, out_buffer, (size_t) out_buffer_size);(*env)->ReleaseByteArrayElements(env, audio_data_byteArray, fp_AudioDataArray,0);// AudioTrack.write PCM数据(*env)->CallIntMethod(env,player->audio_track,player->audio_track_write_mid,audio_data_byteArray, 0, out_buffer_size);//！！！释放局部引用，要不然会局部引用溢出(*env)->DeleteLocalRef(env,audio_data_byteArray);}}av_packet_unref(packet);}end:av_free(out_buffer);av_frame_free(&frame);LOGI("thread_audio_avframe_decoder exit ...\n");(*gJavaVM)->DetachCurrentThread(gJavaVM);return 0;
}

首先，我们通过AVFormatContext->streams[stream_idx]获取得到当前解码的流对象AVStream，然后紧接着，留意中间部分的 // !test 部分代码，我们通过av_frame_get_best_effort_timestamp方法，获取当前解码帧pts的刻度，把当前的pts刻度打印出来，然后乘以一个当前音频流的时间基 av_q2d(audioStream->time_base) 得到真正的 pts，再打印出来。em... 如何理解？？？

首先我们来看看AVStream->time_base是个什么玩意，以及这个av_q2d函数。

/*** Rational number (pair of numerator and denominator).*/
typedef struct AVRational{int num; ///< Numeratorint den; ///< Denominator
} AVRational;AVRational time_base;
/
static inline double av_q2d(AVRational a){return a.num / (double) a.den;
}

然后我们再来了解一下，什么是 PTS刻度。

PTS：Presentation Time Stamp。PTS主要用于度量解码后的视频帧什么时候被显示出来
DTS：Decode Time Stamp。DTS主要是标识读入内存中的bit流在什么时候开始送入解码器中进行解码

也就是pts反映帧什么时候开始显示，dts反映数据流什么时候开始解码；怎么理解这里的“什么时候”呢？如果有某一帧，假设它是第10秒开始显示。那么它的pts是多少呢。是10？还是10s？还是两者都不是。

为了回答这个问题，FFmpeg引入了时间基的概念，也就是time_base。它是用来度量时间的基本单元。
如果把1秒分为25等份，你可以理解就是一把尺，那么每一格表示的就是1/25秒。此时的time_base={1，25}
如果你是把1秒分成90000份，每一个刻度就是1/90000秒，此时的time_base={1，90000}。

这么说应该理解av_frame_get_best_effort_timestamp方法获取的pts刻度及其时间基了吧？我们在视频解码线程的相同位置（avcodec_receive_frame之后，渲染之前）加上同样的test代码。启动demo，随后我自己是得到这样的一份调试日志。

[sync_player.c][video_avframe_decoder][229]: video steam time_base : 1/29970
[sync_player.c][audio_avframe_decoder][143]: audio steam time_base : 1/48000
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 1024
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 2048
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 3072
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 0
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 4096
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 5120
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 6144
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 7168
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 2000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 8192
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 9216
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 10240
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 11264
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 12288
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 13312
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 3000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 14336
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 15360
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 16384
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 17408
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 18432
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 4000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 19456
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 20480
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 21504
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 22528
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 23552
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 24576
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 5000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 25600
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 26624
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 27648
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 28672
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 29696
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 30720
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 31744
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 6000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 32768
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 33792
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 34816
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 35840
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 36864
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 37888
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 7000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 38912
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 39936
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 40960
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 41984
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 43008
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 44032
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 8000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 45056
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 46080
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 47104
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 0
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 48128
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 1
[sync_player.c][audio_avframe_decoder][186]: audio current frame PTS : 49152
[sync_player.c][audio_avframe_decoder][188]: audio current frame PTS : 1
[sync_player.c][video_avframe_decoder][288]: video current frame PTS : 9000
[sync_player.c][video_avframe_decoder][290]: video current frame PTS : 0

从第一二行日志，我们可以清楚的知道，audio / video的时间基是各不相同的。这个时间基其实我们是可以间接初步的推算得出的，捉住时间基的概念去理解（1秒的量化单位），所以音频的时间基就是采样频率（每秒音频采样）的倒数，而视频的时间基就是帧率（每秒显示多少帧）的倒数了。（测试MP4文件，帧率29.97≈30，音频采样44100Hz，双声道AAC）

然后从日志我们可以看到，经过一瞬间后，audio的PTS已经满了1个时间基的刻度，到1s了。而且video还是9000/29970，还不足0.5s，已经出现了音视频不同步的现象了。

而且有没同学觉得（反正我自己是这样觉得了）通过这个av_frame_get_best_effort_timestamp方法获取的pts刻度，转换的时间只到秒级，用于调试还可以，但是用于同步那未免太粗糙了。有没更精确的数值表示呢？答案是有的。我们更新 !test 的代码替换如下：

// audio_avframe_decoder
audioClock = frame->pts * av_q2d(audioStream->time_base);
LOGD(" current audioClock : %f\n", audioClock);// video_avframe_decoder
videoClock = frame->pts * av_q2d(videoStream->time_base);
LOGD(" current audioClock : %f\n", audioClock);

新建两个double类型的全局变量，audioClock / videoClock。我们这次直接使用解码后的frame自带的pts字段，日志输出如下

[sync_player.c][video_avframe_decoder][231]: video steam time_base : 1/29970
[sync_player.c][audio_avframe_decoder][142]: audio steam time_base : 1/48000
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.000000
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.021333
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.042667
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.064000
[sync_player.c][video_avframe_decoder][295]:  current videoClock : 0.000000
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.085333
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.106667
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.128000
[sync_player.c][audio_avframe_decoder][191]:  current audioClock : 0.149333
[sync_player.c][video_avframe_decoder][295]:  current videoClock : 0.033367

已经精确到微妙级别了（1s = 1_000ms = 1_000_000us），可以使用于同步逻辑了。

那么要怎样去同步呢？以demo现在这个情况，audioClock比videoClock要快，在恰当的时间需要放慢脚步等待，videoClock太慢要加快，秉持这个原则，我们在音频解码线程的解码循环中，增加代码：

    while(player->stop_thread_audio_decoder == 0){pthread_mutex_lock(&audioAVPacketButter->mutex);AVPacket* packet = get_read_packet(audioAVPacketButter);pthread_mutex_unlock(&audioAVPacketButter->mutex);//AVPacket->AVFrameret = avcodec_send_packet(audioCodecCtx, packet);if (ret == AVERROR_EOF){av_packet_unref(packet);LOGW("audio_decoder avcodec_send_packet：%d\n", ret);break; // 跳出 while(player->stop_thread_audio_decoder==0)}else if(ret < 0){av_packet_unref(packet);LOGE("audio_decoder avcodec_send_packet：%d\n", ret);continue;}while(ret >= 0){ret = avcodec_receive_frame(audioCodecCtx, frame);if (ret == AVERROR(EAGAIN) ) {//LOGD("audio_decoder avcodec_receive_frame：%d\n", ret);break; // 跳出 while(ret>=0)} else if (ret < 0 || ret == AVERROR_EOF) {LOGW("audio_decoder avcodec_receive_frame：%d %s\n", AVERROR(ret), av_err2str(ret));av_packet_unref(packet);goto end;  //end处进行资源释放等善后处理}// !test start//if ((pts = av_frame_get_best_effort_timestamp(frame)) == AV_NOPTS_VALUE)//    pts = 0;//LOGI("audio current frame PTS : %lld\n",pts);//pts *= av_q2d(audioStream->time_base);//LOGI("audio current frame PTS : %lld\n",pts);// !test endaudioClock = frame->pts * av_q2d(audioStream->time_base);//LOGD(" current audioClock : %f\n", audioClock);if (ret >= 0){swr_convert(player->swr_ctx, &out_buffer, MAX_AUDIO_FRAME_SIZE, (const uint8_t **) frame->data, frame->nb_samples);//获取sample的sizeint out_buffer_size = av_samples_get_buffer_size(NULL, player->out_channel_nb,frame->nb_samples, player->out_sample_fmt, 1);//AudioTrack.write(byte[] int int) 需要byte数组,对应jni的jbyteArray//需要把out_buffer缓冲区数据转成byte数组jbyteArray audio_data_byteArray = (*env)->NewByteArray(env, out_buffer_size);jbyte* fp_AudioDataArray = (*env)->GetByteArrayElements(env, audio_data_byteArray, NULL);memcpy(fp_AudioDataArray, out_buffer, (size_t) out_buffer_size);(*env)->ReleaseByteArrayElements(env, audio_data_byteArray, fp_AudioDataArray,0);// AudioTrack.write PCM数据(*env)->CallIntMethod(env,player->audio_track,player->audio_track_write_mid,audio_data_byteArray, 0, out_buffer_size);if(fabs(audioClock-videoClock) > AV_SYNC_THRESHOLD){   // 再次填充PCM + usleep延迟等待(*env)->CallIntMethod(env,player->audio_track,player->audio_track_write_mid,audio_data_byteArray, 0, out_buffer_size);usleep(15000); // 等待时间其实还需要更好的通过pts计算出来。}//！！！释放局部引用，要不然会局部引用溢出(*env)->DeleteLocalRef(env,audio_data_byteArray);}}av_packet_unref(packet);}

在正常的AudioTrack.write写入PCM数据之后，我们判断当前 audioClock-videoClock的差值，如果大于定义的同步差值，我们就usleep睡眠延迟等待，并重新写入该帧的PCM数据（因为人类的听觉灵敏度远大于视觉灵敏度，变音好过失去声音吧？）以降低audio的速度。

    while(player->stop_thread_video_decoder == 0){pthread_mutex_lock(&videoAVPacketButter->mutex);AVPacket* packet = get_read_packet(videoAVPacketButter);pthread_mutex_unlock(&videoAVPacketButter->mutex);//AVPacket->AVFrameret = avcodec_send_packet(videoCodecCtx, packet);if (ret == AVERROR_EOF){av_packet_unref(packet);LOGW("video_decoder avcodec_send_packet：%d\n", ret);break; //跳出 while(player->stop_thread_video_decoder == 0)}else if(ret < 0){av_packet_unref(packet);LOGE("video_decoder avcodec_send_packet：%d\n", ret);continue;}while(ret >= 0){ret = avcodec_receive_frame(videoCodecCtx, yuv_frame);if (ret == AVERROR(EAGAIN) ){//LOGD("video_decoder avcodec_receive_frame：%d\n", ret);break; //跳出 while(ret >= 0)}else if (ret < 0 || ret == AVERROR_EOF) {LOGW("video_decoder avcodec_receive_frame：%d %s\n", AVERROR(ret),av_err2str(ret));av_packet_unref(packet);goto end;}// !test start//if ((pts = av_frame_get_best_effort_timestamp(yuv_frame)) == AV_NOPTS_VALUE)//    pts = 0;//LOGD("video current frame PTS : %lld\n",pts);//pts *= av_q2d(videoStream->time_base);//LOGD("video current frame PTS : %lld\n",pts);// !test endvideoClock = yuv_frame->pts * av_q2d(videoStream->time_base);//LOGD(" current videoClock : %f\n", videoClock);if(fabs(audioClock-videoClock) > AV_SYNC_THRESHOLD)continue;if (ret >= 0){ANativeWindow_lock(nativeWindow, &nativeWinBuffer, NULL);// 上锁并关联 ANativeWindow + ANativeWindow_Bufferav_image_fill_arrays(rgb_frame->data, rgb_frame->linesize, nativeWinBuffer.bits,AV_PIX_FMT_RGBA, videoCodecCtx->width, videoCodecCtx->height, 1 );// rgb.AVFrame对象 关联 ANativeWindow_Buffer的真实内存空间actual bits.I420ToARGB(yuv_frame->data[0], yuv_frame->linesize[0],yuv_frame->data[2], yuv_frame->linesize[2],yuv_frame->data[1], yuv_frame->linesize[1],rgb_frame->data[0], rgb_frame->linesize[0],videoCodecCtx->width, videoCodecCtx->height);// yuv.AVFrame 转 rgb.AVFrameANativeWindow_unlockAndPost(nativeWindow);// 释放锁并 swap交换显示内存到屏幕上。}}av_packet_unref(packet);}

视频解码线程的循环当中，在avframe正式被渲染到ANativeWindow之前，也进行一下audioClock-videoClock的判断，如果大于定义的同步差值，就直接跳过当前帧的渲染，丢帧的操作是以加快video的速度，追赶audio。

至于这个AV_SYNC_THRESHOLD的值是多少呢？这个是要根据之前的videoClock 和 audioClock的跨度决定，具体多少没有定值，需要我们自己根据实际情况去进行有策略性的调整。

进一步探究

现在demo的音视同步效果其实是差强人意的，但是原理我们已经理解清楚了。接下来，我们就要一探究竟，为何video的速度如此之慢，我们根本达不到 30fps的效果，为何其他播放器就可以呢？我们暂且先屏蔽掉同步的逻辑，在视频解码线程的循环内加入代码耗时时间的调试日志

    int ret;int64_t pts;long start;int count = 0;while(player->stop_thread_video_decoder == 0){count++;start = clock();LOGD("video decoder %d round start at %ld ...\n", count, start/1000);pthread_mutex_lock(&videoAVPacketButter->mutex);AVPacket* packet = get_read_packet(videoAVPacketButter);pthread_mutex_unlock(&videoAVPacketButter->mutex);LOGI("video get_read_packet at %ld \n", clock()-start);//AVPacket->AVFrameret = avcodec_send_packet(videoCodecCtx, packet);LOGI("video avcodec_send_packet at %ld \n", clock()-start);if (ret == AVERROR_EOF){av_packet_unref(packet);LOGW("video_decoder avcodec_send_packet：%d\n", ret);break; //跳出 while(player->stop_thread_video_decoder == 0)}else if(ret < 0){av_packet_unref(packet);LOGE("video_decoder avcodec_send_packet：%d\n", ret);continue;}while(ret >= 0){ret = avcodec_receive_frame(videoCodecCtx, yuv_frame);if (ret == AVERROR(EAGAIN) ){//LOGD("video_decoder avcodec_receive_frame：%d\n", ret);break; //跳出 while(ret >= 0)}else if (ret < 0 || ret == AVERROR_EOF) {LOGW("video_decoder avcodec_receive_frame：%d %s\n", AVERROR(ret),av_err2str(ret));av_packet_unref(packet);goto end;}LOGI("video avcodec_receive_frame at %ld \n", clock()-start);// !test start//if ((pts = av_frame_get_best_effort_timestamp(yuv_frame)) == AV_NOPTS_VALUE)//    pts = 0;//LOGD("video current frame PTS : %lld\n",pts);//pts *= av_q2d(videoStream->time_base);//LOGD("video current frame PTS : %lld\n",pts);// !test endvideoClock = yuv_frame->pts * av_q2d(videoStream->time_base);//LOGD(" current videoClock : %f\n", videoClock);//if(fabs(audioClock-videoClock) > AV_SYNC_THRESHOLD)//    continue;if (ret >= 0){LOGI("video prepare draw at %ld \n", clock()-start);ANativeWindow_lock(nativeWindow, &nativeWinBuffer, NULL);// 上锁并关联 ANativeWindow + ANativeWindow_Bufferav_image_fill_arrays(rgb_frame->data, rgb_frame->linesize, nativeWinBuffer.bits,AV_PIX_FMT_RGBA, videoCodecCtx->width, videoCodecCtx->height, 1 );// rgb.AVFrame对象 关联 ANativeWindow_Buffer的真实内存空间actual bits.I420ToARGB(yuv_frame->data[0], yuv_frame->linesize[0],yuv_frame->data[2], yuv_frame->linesize[2],yuv_frame->data[1], yuv_frame->linesize[1],rgb_frame->data[0], rgb_frame->linesize[0],videoCodecCtx->width, videoCodecCtx->height);LOGI("video I420ToARGB at %ld \n", clock()-start);// yuv.AVFrame 转 rgb.AVFrameANativeWindow_unlockAndPost(nativeWindow);// 释放锁并 swap交换显示内存到屏幕上。LOGI("video finish draw at %ld \n", clock()-start);}}av_packet_unref(packet);LOGD("video decoder %d round end at %ld ...\n", count, (clock()-start)/1000);LOGD("-------------------------------------\n");}

我们在各个关键位置，加入了耗时的打印。得出如下日志。

[sync_player.c][video_avframe_decoder][268]: video decoder 1 round start at 1223 ...
[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 53
[sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 39951
[sync_player.c][video_avframe_decoder][336]: video decoder 1 round end at 40036 ...
[sync_player.c][video_avframe_decoder][337]: -------------------------------------
[sync_player.c][video_avframe_decoder][268]: video decoder 2 round start at 1263 ...
[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 25
[sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 28174
[sync_player.c][video_avframe_decoder][298]: video avcodec_receive_frame at 28257
[sync_player.c][video_avframe_decoder][317]: video prepare draw at 28290
[sync_player.c][video_avframe_decoder][328]: video I420ToARGB at 115386
[sync_player.c][video_avframe_decoder][332]: video finish draw at 116751
[sync_player.c][video_avframe_decoder][336]: video decoder 2 round end at 116843 ...
[sync_player.c][video_avframe_decoder][337]: -------------------------------------
[sync_player.c][video_avframe_decoder][268]: video decoder 3 round start at 1380 ...
[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 44
[sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 16678
[sync_player.c][video_avframe_decoder][298]: video avcodec_receive_frame at 16788
[sync_player.c][video_avframe_decoder][317]: video prepare draw at 16831
[sync_player.c][video_avframe_decoder][328]: video I420ToARGB at 90281
[sync_player.c][video_avframe_decoder][332]: video finish draw at 91667
[sync_player.c][video_avframe_decoder][336]: video decoder 3 round end at 91758 ...
[sync_player.c][video_avframe_decoder][337]: -------------------------------------

分析前三round，我们已经可以明显的知道了。每一round end得出的耗时大头，就是I420ToARGB方法就是yuv转rgb的整个耗时时间，这平均耗时居然达100ms，视频的帧率变成10fps。（笑哭.jpg）
那么有同学可能会说，不要用libyuv了，直接用ffmpeg的SwsContext API转换格式。效率会不会好一点？这个随便百度一下，可以知道libyuv比ffmpeg自带的转换格式性能提高，是接近10倍哦。
流弊一点的同学可能就会低头推眼镜说：麻烦加上neon指令加速。是的，NEON确实也能提高其中的效率，效果怎样暂且我还没试验，网上论调也不尽人意。（但是开启NEON加速总比没有开启的好啊！）

那怎么办？这效果boss不接受啊！用户更加不能忍啊！既然yuv转rgb耗时那么多，为啥不能把这部分砍掉？直接用yuv渲染？是的，已经有前人想到了这点，并从前期的Android系统的播放器找到了答案，但是代码过于复杂而且年代久远，Google也放弃了AwesomePlayer。用于学习还是可以的。

Android用surface直接显示yuv数据（一）

Android用surface直接显示yuv数据（二）

Android用surface直接显示yuv数据（三）

然后我想说的是！OpenGL也能轻易的做到这一点！所以我会在下个主题OpenGL+Shader的学习中重回这里，让我们的demo真正的做到30fps，甚至更高的60fps！

2019.01.24：升级libyuv支持neon，性能得到提升。

文章当天写完之后，我就开始着手升级ffmpeg支持neon和libyuv支持neon的工作。其实两个库支持neon都比较简单。

ffmpeg支持neon，只需要在之前手动编译Android使用的FFmpeg库（Linux）介绍的脚本，--enable-neon就可以了。

#!/bin/bash
make clean
export NDK=/usr/ndk/android-ndk-r14/
export SYSROOT=$NDK/platforms/android-9/arch-arm/
export TOOLCHAIN=$NDK/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64
export CPU=arm
export PREFIX=$(pwd)/android/$CPU
export ADDI_CFLAGS="-marm"./configure --target-os=linux \
--prefix=$PREFIX --arch=arm \
--disable-doc \
--enable-shared \
--disable-static \
--disable-yasm \
--disable-symver \
--enable-gpl \
--enable-neon \
--disable-ffmpeg \
--disable-ffplay \
--disable-ffprobe \
--disable-ffserver \
--disable-doc \
--disable-symver \
--cross-prefix=$TOOLCHAIN/bin/arm-linux-androideabi- \
--enable-cross-compile \
--sysroot=$SYSROOT \
--extra-cflags="-Os -fpic $ADDI_CFLAGS" \
--extra-ldflags="$ADDI_LDFLAGS" \
$ADDITIONAL_CONFIGURE_FLAG
make clean
make
make install

libyuv之前好像没介绍源码编译的过程，因为都比较简单。参考https://blog.csdn.net/abcdnml/article/details/70243783就可以了。

LOCAL_PATH:= $(call my-dir)
include $(CLEAR_VARS)LOCAL_CPP_EXTENSION := .cc
LOCAL_SRC_FILES := \source/compare.cc           \source/compare_common.cc    \source/compare_neon64.cc    \source/compare_gcc.cc       \source/convert.cc           \source/convert_argb.cc      \source/convert_from.cc      \source/convert_from_argb.cc \source/convert_to_argb.cc   \source/convert_to_i420.cc   \source/cpu_id.cc            \source/planar_functions.cc  \source/rotate.cc            \source/rotate_argb.cc       \source/rotate_mips.cc       \source/rotate_neon64.cc     \source/row_any.cc           \source/row_common.cc        \source/row_mips.cc          \source/row_neon64.cc        \source/row_gcc.cc           \source/scale.cc             \source/scale_any.cc         \source/scale_argb.cc        \source/scale_common.cc      \source/scale_mips.cc        \source/scale_neon64.cc      \source/scale_gcc.cc         \source/video_common.cc# TODO(fbarchard): Enable mjpeg encoder.
#   source/mjpeg_decoder.cc
#   source/convert_jpeg.cc
#   source/mjpeg_validate.ccifeq ($(TARGET_ARCH_ABI),armeabi-v7a)LOCAL_CFLAGS += -DLIBYUV_NEONLOCAL_SRC_FILES += \source/compare_neon.cc.neon    \source/rotate_neon.cc.neon     \source/row_neon.cc.neon        \source/scale_neon.cc.neon
endifLOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/include
LOCAL_C_INCLUDES += $(LOCAL_PATH)/includeLOCAL_MODULE := libyuv
LOCAL_MODULE_TAGS := optionalinclude $(BUILD_SHARED_LIBRARY)

我们可以看到，libyuv的编译脚本，其实在TARGET_ARCH_ABI==armeabi-v7a的时候，已经自动的增加了neon的支持。所以我们必须要另外写Application.mk，指定编译平台APP_ABI := armeabi armeabi-v7a mips，然后使用lib/armeabi-v7a的so。

经过升级以后，我们在看看解码线程的调试日志。

 [sync_player.c][audio_avframe_decoder][140]: audio steam time_base : 1/48000 [sync_player.c][video_avframe_decoder][235]: video steam time_base : 1/29970 [sync_player.c][video_avframe_decoder][268]: video decoder 1 round start at 1190 ...[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 78 [sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 39669 [sync_player.c][video_avframe_decoder][335]: video decoder 1 round end at 39 ...[sync_player.c][video_avframe_decoder][336]: -------------------------------------[sync_player.c][video_avframe_decoder][268]: video decoder 2 round start at 1230 ...[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 23 [sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 39946 [sync_player.c][video_avframe_decoder][298]: video avcodec_receive_frame at 40033 [sync_player.c][video_avframe_decoder][335]: video decoder 2 round end at 40 ...[sync_player.c][video_avframe_decoder][336]: -------------------------------------[sync_player.c][video_avframe_decoder][268]: video decoder 3 round start at 1270 ...[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 23 [sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 15373 [sync_player.c][video_avframe_decoder][298]: video avcodec_receive_frame at 15459 [sync_player.c][video_avframe_decoder][317]: video prepare draw at 15497 [sync_player.c][video_avframe_decoder][331]: video finish draw at 25104 [sync_player.c][video_avframe_decoder][335]: video decoder 3 round end at 25 ...[sync_player.c][video_avframe_decoder][336]: -------------------------------------[sync_player.c][video_avframe_decoder][268]: video decoder 4 round start at 1295 ...[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 23 [sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 38360 [sync_player.c][video_avframe_decoder][298]: video avcodec_receive_frame at 38448 [sync_player.c][video_avframe_decoder][335]: video decoder 4 round end at 38 ...[sync_player.c][video_avframe_decoder][336]: -------------------------------------[sync_player.c][video_avframe_decoder][268]: video decoder 5 round start at 1334 ...[sync_player.c][video_avframe_decoder][272]: video get_read_packet at 23 [sync_player.c][video_avframe_decoder][276]: video avcodec_send_packet at 49854 [sync_player.c][video_avframe_decoder][298]: video avcodec_receive_frame at 49948 [sync_player.c][video_avframe_decoder][317]: video prepare draw at 49990 [sync_player.c][video_avframe_decoder][331]: video finish draw at 58722 [sync_player.c][video_avframe_decoder][335]: video decoder 5 round end at 58 ...[sync_player.c][video_avframe_decoder][336]: -------------------------------------

比较video finish - prepare的差值，解码线程的性能提高了10倍。