最近在side project中遇到了需要从视频中抓取多张图片的需求。安卓已经提供了从视频获取预览图片的ThumbnailUtils, 但此类不能根据timestamp获取bitmap。

以下记录自己找出的解决方案。

需求

在本地视频根据时间戳(timestamp)抓取bitmap图像。

解决方案

配合使用这些flag可以达到不同的程度的时间精确度

OPTION_PREVIOUS_SYNC： 前一个i-frame

OPTION_NEXT_SYNC： 后一个 i-frame

OPTION_CLOSEST_SYNC： 最近的i-frame，不管前后

OPTION_CLOSEST： 最近的frame，不一定是i-frame。上面三个flag只需要decode一张frame即可(而且是i-frame)。但这个flag需要decode多张frame才能接近输入的timestamp，因此速度会慢些。

值得一提的是此类会用binder IBC给system media service发送请求， 因此decoder是在系统服务中进行的，并非在我自己的app进程中。

方法调用时logcat可看到如下log

918 13498 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process

918 28062 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process

918 1854 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process

918 13498 I OMXMaster: makeComponentInstance(OMX.google.h264.decoder) in omx@1.0-service process

需要注意的坑

如果只看文档就会以为用getFrameAtTime + OPTION_CLOSEST就能非常精准的返回在输入timestamp附近的视频帧，但在不同device上跑过代码才发现其行为其实很不统一。:(

高端机基本都能做到预期效果，返回的bitmap在输入timestamp附近。但低端机型直接无视OPTION_CLOSEST 返回附近I-frame的bitmap，因此有时得到的图像跟输入的timestamp相差甚远。例如输入的是3.3sec，返回的有可能是2.8sec的i-frame图像。

出于好奇，我在低端机上做了个小实验。扫描一个视频文件的video timestamp，再把他们逐一用来调用getFrameAtTime，看看返回的bitmap有没有重复。

很明显的看到高端机基本无重复的bitmap，低端机却有大把大把的重复。猜测应该是厂商为了性能而只做了附近i-frame的解码就返回了。有点坑。：(

如果需要在所有机器上返回精准timestamp的bitmap，恐怕只能自己操作decoder了。估计非常复杂。

此外 API28+ 新增加了MediaMetadataRetriever.getFramesAtIndex() 。 这应该是比pts更加稳定的抓取方式，毕竟index是连续的int。但不知低端机型会不会继续坑。

==高端

D/MediaMetadataRetrieverRunner: pts 33375 digest :ym71cu9iO1H94190FWVgeg==

D/MediaMetadataRetrieverRunner: pts 66625 digest :y9hG93MqABw3ILMta/noUg==

D/MediaMetadataRetrieverRunner: pts 100000 digest :0l2CkOYDNyxvZCfCeLuq9A==

D/MediaMetadataRetrieverRunner: pts 133375 digest :OHSVQQOxrO77mh9f7tTorQ==

D/MediaMetadataRetrieverRunner: pts 166625 digest :XZ8kDFxsdjgBkUzKl41dgA==

D/MediaMetadataRetrieverRunner: pts 200000 digest :0/13IPmw1gP9FLdpsJc/Wg==

D/MediaMetadataRetrieverRunner: pts 233375 digest :TeHChk9OWbb8nLbrFphlEg==

D/MediaMetadataRetrieverRunner: pts 266625 digest :GEzE3LtFzG88bT7WtMTbxA==

D/MediaMetadataRetrieverRunner: pts 300000 digest :QjdeoZYwYgNxV144kKLTNw==

D/MediaMetadataRetrieverRunner: pts 333375 digest :0DZCQer3BqMPfSYTJaafMg==

D/MediaMetadataRetrieverRunner: pts 366625 digest :mfjEap/awknngWIeyPAScg==

D/MediaMetadataRetrieverRunner: pts 400000 digest :Q1oSKau7n9boi0vDSFZJFA==

D/MediaMetadataRetrieverRunner: pts 433375 digest :kW0vjMra+boztvH+PXQrlw==

D/MediaMetadataRetrieverRunner: pts 466625 digest :a6C/BpHAcr4BxSDg7Gt4bQ==

D/MediaMetadataRetrieverRunner: pts 500000 digest :odAEN5ESKWrLmXwDu09arA==

D/MediaMetadataRetrieverRunner: pts 533375 digest :+zTTQsil/s6on0EVqFuH/Q==

D/MediaMetadataRetrieverRunner: pts 566625 digest :ZqsNHZd1r12UL7cFHdc9vw==

D/MediaMetadataRetrieverRunner: pts 600000 digest :CgUZWadeCe+S+e28C/4qkA==

D/MediaMetadataRetrieverRunner: pts 633375 digest :hUaJy2jWWa9RU4dR24Om7w==

D/MediaMetadataRetrieverRunner: pts 666625 digest :a5HlUYUJcz3xaQrn/hNsIg==

D/MediaMetadataRetrieverRunner: pts 700000 digest :YF70StAojixZxjk8epZL3w==

D/MediaMetadataRetrieverRunner: pts 733375 digest :rolF61sMxRMSP09ePtUGcQ==

D/MediaMetadataRetrieverRunner: pts 766625 digest :K+qZUFNP5EgzWPscmmnFew==

D/MediaMetadataRetrieverRunner: pts 800000 digest :d8iVGhHf3VpMn+vMnBdmng==

D/MediaMetadataRetrieverRunner: pts 833375 digest :cKREwaki8AJmOuSVa8Zvbw==

D/MediaMetadataRetrieverRunner: pts 866625 digest :b3QXkX+wTE1CuCe79JK7Ww==

D/MediaMetadataRetrieverRunner: pts 900000 digest :atzAyrZNcOf8Ghgf04lftw==

D/MediaMetadataRetrieverRunner: pts 933375 digest :hS9ukOCLMCobHplBeRNdOA==

D/MediaMetadataRetrieverRunner: pts 966625 digest :rZyr6vVt5ae+TiMMVTCRrg==

=====

==低端

D/MediaMetadataRetrieverRunner: pts 1601625 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1635000 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1668375 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1701750 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1735000 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1768375 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1801750 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1835125 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1868500 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1901875 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1935250 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 1968625 digest :Ba29dyKehU1o6EanhG0wvg==

D/MediaMetadataRetrieverRunner: pts 2002000 digest :dcMXdpTton6/YQHJA2zCjg==

D/MediaMetadataRetrieverRunner: pts 2035375 digest :830odrRgw9UyAhNKeBUWAA==

D/MediaMetadataRetrieverRunner: pts 2068750 digest :8WFabowQKX+I2q3XNC2HJg==

D/MediaMetadataRetrieverRunner: pts 2102125 digest :8WFabowQKX+I2q3XNC2HJg==

D/MediaMetadataRetrieverRunner: pts 2135500 digest :8WFabowQKX+I2q3XNC2HJg==

D/MediaMetadataRetrieverRunner: pts 2168875 digest :dcMXdpTton6/YQHJA2zCjg==

D/MediaMetadataRetrieverRunner: pts 2202250 digest :dcMXdpTton6/YQHJA2zCjg==

D/MediaMetadataRetrieverRunner: pts 2235500 digest :dcMXdpTton6/YQHJA2zCjg==

D/MediaMetadataRetrieverRunner: pts 2268875 digest :dcMXdpTton6/YQHJA2zCjg==

D/MediaMetadataRetrieverRunner: pts 2302250 digest :dcMXdpTton6/YQHJA2zCjg==

D/MediaMetadataRetrieverRunner: pts 2335625 digest :h5ZNxcxO6WrbYFQPsknjnw==

D/MediaMetadataRetrieverRunner: pts 2369000 digest :KXZQ788bP25Oqi7sHxWPLg==

D/MediaMetadataRetrieverRunner: pts 2402375 digest :KXZQ788bP25Oqi7sHxWPLg==

D/MediaMetadataRetrieverRunner: pts 2435750 digest :KXZQ788bP25Oqi7sHxWPLg==

D/MediaMetadataRetrieverRunner: pts 2469125 digest :KXZQ788bP25Oqi7sHxWPLg==

D/MediaMetadataRetrieverRunner: pts 2502500 digest :2h341j90Scn3kRtA4Fr+IA==

D/MediaMetadataRetrieverRunner: pts 2535875 digest :2h341j90Scn3kRtA4Fr+IA==

D/MediaMetadataRetrieverRunner: pts 2569250 digest :OOdPxTb4/NWbUrC8HDzpng==

D/MediaMetadataRetrieverRunner: pts 2602625 digest :OOdPxTb4/NWbUrC8HDzpng==

D/MediaMetadataRetrieverRunner: pts 2636000 digest :OOdPxTb4/NWbUrC8HDzpng==

D/MediaMetadataRetrieverRunner: pts 2669375 digest :OOdPxTb4/NWbUrC8HDzpng==

D/MediaMetadataRetrieverRunner: pts 2702750 digest :OOdPxTb4/NWbUrC8HDzpng==

相关代码

public class MediaMetadataRetrieverRunner {

private String TAG = MediaMetadataRetrieverRunner.class.getSimpleName();

void scanAndDigest(String path) throws Exception {

List ptsUsList = VideoPtsScanner.scanPts(path);

Log.d(TAG, "pts list in micro sec: " + ptsUsList);

MediaMetadataRetriever retriever = new MediaMetadataRetriever();

retriever.setDataSource(path);

for (long ptsUs : ptsUsList) {

Bitmap bitmap = retriever.getFrameAtTime(ptsUs, MediaMetadataRetriever.OPTION_CLOSEST);

if (bitmap == null) {

Log.e(TAG, "got null in pts " + ptsUs);

continue;

}

int size = bitmap.getAllocationByteCount();

ByteBuffer buffer = ByteBuffer.allocateDirect(size);

bitmap.copyPixelsToBuffer(buffer);

buffer.flip();

String digest = makeString(buffer);

Log.d(TAG, "pts " + ptsUs + " \t digest :" + digest);

}

retriever.release();

}

private String makeString(ByteBuffer buffer) throws NoSuchAlgorithmException {

MessageDigest md5 = MessageDigest.getInstance("MD5");

md5.update(buffer);

byte[] digest = md5.digest();

// covert into base64 string, since it's already provided in android

return Base64.encodeToString(digest, Base64.DEFAULT);

}

}

-----

public class VideoPtsScanner {

public static List scanPts(String path) throws IOException {

MediaExtractor mediaExtractor = new MediaExtractor();

mediaExtractor.setDataSource(path);

mediaExtractor.selectTrack(getVideoTrack(mediaExtractor));

List ret = new ArrayList();

long pts = -1;

while ((pts = mediaExtractor.getSampleTime()) >= 0) {

ret.add(pts);

mediaExtractor.advance();

}

mediaExtractor.release();

return ret;

}

private static int getVideoTrack(MediaExtractor extractor) {

for (int i = 0; i < extractor.getTrackCount(); i++) {

MediaFormat format = extractor.getTrackFormat(i);

if (isVideo(format)) {

return i;

}

}

return -1;

}

private static boolean isVideo(MediaFormat format) {

return format.getString(MediaFormat.KEY_MIME).toLowerCase().contains("video");

}

}