Android本身提供了音視頻編解碼工具，很多時(shí)候是不需要第三方工具的，比如ffmpeg, OpenCV等，在android中引入第三庫比較復(fù)雜，在Android音視頻編碼中介紹了如何引入第三方庫libpng來進(jìn)行進(jìn)行圖片處理，同時(shí)引入這些第三方庫，是程序結(jié)構(gòu)變得復(fù)雜。
本文介紹的音視頻編解碼利用的就是android自帶的MediaCodec。視頻編碼之后，你可以對(duì)視頻做任何形式的處理，比如添加廣告，剪輯等等。
MediaCodec
官方文檔： MediaCodec
MediaCodec class can be used to access low-level media codecs, i.e. encoder/decoder components. It is part of the Android low-level multimedia support infrastructure (normally used together with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack.)
在其生命周期中，編解碼器概念上存在三種狀態(tài)：停止、執(zhí)行和釋放。停止?fàn)顟B(tài)實(shí)際上是三種狀態(tài)的集合：未初始化、已配置和錯(cuò)誤。而執(zhí)行狀態(tài)在概念上分為三個(gè)子狀態(tài)：刷新、運(yùn)行和流結(jié)束。
使用Buffers進(jìn)行異步處理
Since Build.VERSION_CODES.LOLLIPOP, the preferred method is to process data asynchronously by setting a callback before calling configure.
從Android5.0之后，谷歌就建議使用異步的方式。
MediaCodec codec = MediaCodec.createByCodecName(name);
MediaFormat mOutputFormat; // member variable
codec.setCallback(new MediaCodec.Callback() {
@Override
void onInputBufferAvailable(MediaCodec mc, int inputBufferId) {
ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferId);
// fill inputBuffer with valid data
…
codec.queueInputBuffer(inputBufferId, …);
}
@Override
void onOutputBufferAvailable(MediaCodec mc, int outputBufferId, …) {
ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
// bufferFormat is equivalent to mOutputFormat
// outputBuffer is ready to be processed or rendered.
…
codec.releaseOutputBuffer(outputBufferId, …);
}
@Override
void onOutputFormatChanged(MediaCodec mc, MediaFormat format) {
// Subsequent data will conform to new format.
// Can ignore if using getOutputFormat(outputBufferId)
mOutputFormat = format; // option B
}
@Override
void onError(…) {
…
}
@Override
void onCryptoError(…) {
…
}
});
codec.configure(format, …);
mOutputFormat = codec.getOutputFormat(); // option B
codec.start();
// wait for processing to complete
codec.stop();
codec.release();
Since Build.VERSION_CODES.LOLLIPOP, you should retrieve input and output buffers using getInput/OutputBuffer(int) and/or getInput/OutputImage(int) even when using the codec in synchronous mode
在5.0之前，使用同步的方式。通過 getInput/OutputBuffer(int) 或者getInput/OutputImage(int) 來獲取輸入輸出流。
MediaCodec codec = MediaCodec.createByCodecName(name);
codec.configure(format, …);
MediaFormat outputFormat = codec.getOutputFormat(); // option B
codec.start();
for (;;) {
int inputBufferId = codec.dequeueInputBuffer(timeoutUs);
if (inputBufferId >= 0) {
ByteBuffer inputBuffer = codec.getInputBuffer(…);
// fill inputBuffer with valid data
…
codec.queueInputBuffer(inputBufferId, …);
}
int outputBufferId = codec.dequeueOutputBuffer(…);
if (outputBufferId >= 0) {
ByteBuffer outputBuffer = codec.getOutputBuffer(outputBufferId);
MediaFormat bufferFormat = codec.getOutputFormat(outputBufferId); // option A
// bufferFormat is identical to outputFormat
// outputBuffer is ready to be processed or rendered.
…
codec.releaseOutputBuffer(outputBufferId, …);
} else if (outputBufferId == MediaCodec.INFO_OUTPUT_FORMAT_CHANGED) {
// Subsequent data will conform to new format.
// Can ignore if using getOutputFormat(outputBufferId)
outputFormat = codec.getOutputFormat(); // option B
}
}
codec.stop();
codec.release();
音視頻編解碼
這里使用同步編碼，異步解碼。
使用場景是需要截取某一段視頻某個(gè)時(shí)間段的內(nèi)容，則需要首先對(duì)原視頻進(jìn)行解碼，獲取相應(yīng)時(shí)間段的開始幀和結(jié)束幀率。然后根據(jù)從開始幀和結(jié)束幀對(duì)視頻進(jìn)行重新編碼。
關(guān)于視頻幀
I幀又稱幀內(nèi)編碼幀，是一種自帶全部信息的獨(dú)立幀，無需參考其他圖像便可獨(dú)立進(jìn)行解碼，可以簡單理解為一張靜態(tài)畫面。視頻序列中的第一個(gè)幀始終都是I幀，因?yàn)樗?strong>關(guān)鍵幀。
P幀又稱幀間預(yù)測編碼幀，需要參考前面的I幀才能進(jìn)行編碼。表示的是當(dāng)前幀畫面與前一幀（前一幀可能是I幀也可能是P幀）的差別
B幀又稱雙向預(yù)測編碼幀，也就是B幀記錄的是本幀與前后幀的差別。也就是說要解碼B幀，不僅要取得之前的緩存畫面，還要解碼之后的畫面，通過前后畫面的與本幀數(shù)據(jù)的疊加取得最終的畫面。
這里直接使用獲取的幀，可能是i,p,b幀，根據(jù)場景需要可以自己修改。
同步音視頻編碼
public class VideoEncoder {
private static final String TAG = "VideoEncoder";
private final static String MIME_TYPE = MediaFormat.MIMETYPE_VIDEO_AVC;
private static final long DEFAULT_TIMEOUT_US = 10000;
private MediaCodec mEncoder;
private MediaMuxer mMediaMuxer;
private int mVideoTrackIndex;
private boolean mStop = false;
private AudioEncode mAudioEncode;
public void init(String outPath, int width, int height) {
try {
mStop = false;
mVideoTrackIndex = -1;
mMediaMuxer = new MediaMuxer(outPath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
mEncoder = MediaCodec.createEncoderByType(MIME_TYPE);
MediaFormat mediaFormat = MediaFormat.createVideoFormat(MIME_TYPE, width, height);
mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420SemiPlanar);
mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, width * height * 6);
mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 30);
mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
mEncoder.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
mEncoder.start();
} catch (Throwable e) {
e.printStackTrace();
}
}
public void release() {
mStop = true;
if (mEncoder != null) {
mEncoder.stop();
mEncoder.release();
mEncoder = null;
}
if (mMediaMuxer != null) {
mMediaMuxer.stop();
mMediaMuxer.release();
mMediaMuxer = null;
}
if (mAudioEncode != null) {
mAudioEncode.release();
}
}
public void encode(byte[] yuv, long presentationTimeUs) {
if (mEncoder == null || mMediaMuxer == null) {
Log.e(TAG, "mEncoder or mMediaMuxer is null");
return;
}
if (yuv == null) {
Log.e(TAG, "input yuv data is null");
return;
}
int inputBufferIndex = mEncoder.dequeueInputBuffer(DEFAULT_TIMEOUT_US);
Log.d(TAG, "inputBufferIndex: " + inputBufferIndex);
if (inputBufferIndex == -1) {
Log.e(TAG, "no valid buffer available");
return;
}
ByteBuffer inputBuffer = mEncoder.getInputBuffer(inputBufferIndex);
inputBuffer.put(yuv);
mEncoder.queueInputBuffer(inputBufferIndex, 0, yuv.length, presentationTimeUs, 0);
while (!mStop) {
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
int outputBufferIndex = mEncoder.dequeueOutputBuffer(bufferInfo, DEFAULT_TIMEOUT_US);
Log.d(TAG, "outputBufferIndex: " + outputBufferIndex);
if (outputBufferIndex >= 0) {
ByteBuffer outputBuffer = mEncoder.getOutputBuffer(outputBufferIndex);
// write head info
if (mVideoTrackIndex == -1) {
Log.d(TAG, "this is first frame, call writeHeadInfo first");
mVideoTrackIndex = writeHeadInfo(outputBuffer, bufferInfo);
}
if ((bufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) == 0) {
Log.d(TAG, "write outputBuffer");
mMediaMuxer.writeSampleData(mVideoTrackIndex, outputBuffer, bufferInfo);
}
mEncoder.releaseOutputBuffer(outputBufferIndex, false);
break; // 跳出循環(huán)
}
}
}
public void encodeAudio() {
//音頻混合
mAudioEncode.encodeAudio();
release();
}
private int writeHeadInfo(ByteBuffer outputBuffer, MediaCodec.BufferInfo bufferInfo) {
byte[] csd = new byte[bufferInfo.size];
outputBuffer.limit(bufferInfo.offset + bufferInfo.size);
outputBuffer.position(bufferInfo.offset);
outputBuffer.get(csd);
ByteBuffer sps = null;
ByteBuffer pps = null;
for (int i = bufferInfo.size - 1; i > 3; i--) {
if (csd[i] == 1 && csd[i - 1] == 0 && csd[i - 2] == 0 && csd[i - 3] == 0) {
sps = ByteBuffer.allocate(i - 3);
pps = ByteBuffer.allocate(bufferInfo.size - (i - 3));
sps.put(csd, 0, i - 3).position(0);
pps.put(csd, i - 3, bufferInfo.size - (i - 3)).position(0);
}
}
MediaFormat outputFormat = mEncoder.getOutputFormat();
if (sps != null && pps != null) {
outputFormat.setByteBuffer("csd-0", sps);
outputFormat.setByteBuffer("csd-1", pps);
}
int videoTrackIndex = mMediaMuxer.addTrack(outputFormat);
mAudioEncode = new AudioEncode(mMediaMuxer);
Log.d(TAG, "videoTrackIndex: " + videoTrackIndex);
mMediaMuxer.start();
return videoTrackIndex;
}
}
音頻編碼：文章來源：http://www.zghlxwxcb.cn/news/detail-796924.html
public class AudioEncode {
private MediaMuxer mediaMuxer;
private int audioTrack;
private MyExtractor audioExtractor;
private MediaFormat audioFormat;
public AudioEncode(MediaMuxer mediaMuxer) {
audioExtractor = new MyExtractor(Constants.VIDEO_PATH);
audioFormat = audioExtractor.getAudioFormat();
if (audioFormat != null) {
audioTrack = mediaMuxer.addTrack(audioFormat);
}
this.mediaMuxer = mediaMuxer;
}
@SuppressLint("WrongConstant")
public void encodeAudio() {
MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
ByteBuffer buffer = ByteBuffer.allocate(1024 * 1024);
if (audioFormat != null) {
//寫完視頻，再把音頻混合進(jìn)去
int audioSize = 0;
//讀取音頻幀的數(shù)據(jù)，直到結(jié)束
while ((audioSize = audioExtractor.readBuffer(buffer, false)) > 0) {
bufferInfo.offset = 0;
bufferInfo.size = audioSize;
bufferInfo.presentationTimeUs = audioExtractor.getSampleTime();
bufferInfo.flags = audioExtractor.getSampleFlags();
mediaMuxer.writeSampleData(audioTrack, buffer, bufferInfo);
}
}
}
public void release() {
this.audioExtractor.release();
}
}
異步音視頻解碼
要使用sleepRender進(jìn)行時(shí)間同步。文章來源地址http://www.zghlxwxcb.cn/news/detail-796924.html
/**
* describe：異步解碼
*/
public class AsyncVideoDecode extends BaseAsyncDecode {
private static final String TAG = "AsyncVideoDecode";
private Surface mSurface;
private long mTime = -1;
private int mOutputFormat = Constants.COLOR_FORMAT_NV12;
private byte[] mYuvBuffer;
private Map<Integer, MediaCodec.BufferInfo> map =
new ConcurrentHashMap<>();
private boolean writeAudioFlag = false;
public AsyncVideoDecode(SurfaceTexture surfaceTexture, long progress) {
super(progress);
mSurface = new Surface(surfaceTexture);
}
private ByteArrayOutputStream outStream = new ByteArrayOutputStream();
static VideoEncoder mVideoEncoder = null;
@Override
public void start(){
super.start();
mediaCodec.setCallback(new MediaCodec.Callback() {
@Override
public void onInputBufferAvailable(@NonNull MediaCodec codec, int index) {
ByteBuffer inputBuffer = mediaCodec.getInputBuffer(index);
int size = extractor.readBuffer(inputBuffer, true);
long st = extractor.getSampleTime();
if (size >= 0) {
codec.queueInputBuffer(
index,
0,
size,
st,
extractor.getSampleFlags()
);
} else {
//結(jié)束
codec.queueInputBuffer(
index,
0,
0,
0,
BUFFER_FLAG_END_OF_STREAM
);
}
}
@Override
public void onOutputBufferAvailable(@NonNull MediaCodec codec, int index, @NonNull MediaCodec.BufferInfo info) {
Message msg = new Message();
msg.what = MSG_VIDEO_OUTPUT;
Bundle bundle = new Bundle();
bundle.putInt("index",index);
bundle.putLong("time",info.presentationTimeUs);
if (info.flags == BUFFER_FLAG_END_OF_STREAM) {
if (mVideoEncoder != null) {
mVideoEncoder.encodeAudio();
}
}
//當(dāng)MediaCodeC配置了輸出Surface時(shí)，此值返回null
Image image = codec.getOutputImage(index);
Rect rect = image.getCropRect();
// if (image != null) {
// Rect rect = null;
// try {
// rect = image.getCropRect();
// } catch (Exception e) {
// throw new RuntimeException(e);
// }
if (mYuvBuffer == null) {
mYuvBuffer = new byte[rect.width()*rect.height()*3/2];
}
YuvImage yuvImage = new YuvImage(ConverUtils.getDataFromImage(image), ImageFormat.NV21, rect.width(), rect.height(), null);
yuvImage.compressToJpeg(rect, 100, outStream);
byte[] bytes = outStream.toByteArray();
if (mVideoEncoder == null) {
mVideoEncoder = new VideoEncoder();
mVideoEncoder.init(Constants.OUTPUT_PATH, rect.width(), rect.height());
}
getDataFromImage(image, mOutputFormat, rect.width(), rect.height());
// mVideoEncoder.encode(ConverUtils.getDataFromImage(image), info.presentationTimeUs);
mVideoEncoder.encode(mYuvBuffer, info.presentationTimeUs);
// }
// ByteBuffer outputBuffer = codec.getOutputBuffer(index);
// ConverUtils.convertByteBufferToBitmap(outputBuffer);
msg.setData(bundle);
mHandler.sendMessage(msg);
}
@Override
public void onError(@NonNull MediaCodec codec, @NonNull MediaCodec.CodecException e) {
codec.stop();
}
@Override
public void onOutputFormatChanged(@NonNull MediaCodec codec, @NonNull MediaFormat format) {
}
});
// mediaCodec.configure(mediaFormat,mSurface,null,0);
mediaCodec.configure(mediaFormat,null,null,0);
mediaCodec.start();
}
private void getDataFromImage(Image image, int colorFormat, int width, int height) {
Rect crop = image.getCropRect();
int format = image.getFormat();
Log.d(TAG, "crop width: " + crop.width() + ", height: " + crop.height());
Image.Plane[] planes = image.getPlanes();
byte[] rowData = new byte[planes[0].getRowStride()];
int channelOffset = 0;
int outputStride = 1;
for (int i = 0; i < planes.length; i++) {
switch (i) {
case 0:
channelOffset = 0;
outputStride = 1;
break;
case 1:
if (colorFormat == Constants.COLOR_FORMAT_I420) {
channelOffset = width * height;
outputStride = 1;
} else if (colorFormat == Constants.COLOR_FORMAT_NV21) {
channelOffset = width * height + 1;
outputStride = 2;
} else if (colorFormat == Constants.COLOR_FORMAT_NV12) {
channelOffset = width * height;
outputStride = 2;
}
break;
case 2:
if (colorFormat == Constants.COLOR_FORMAT_I420) {
channelOffset = (int) (width * height * 1.25);
outputStride = 1;
} else if (colorFormat ==Constants.COLOR_FORMAT_NV21) {
channelOffset = width * height;
outputStride = 2;
} else if (colorFormat == Constants.COLOR_FORMAT_NV12) {
channelOffset = width * height + 1;
outputStride = 2;
}
break;
default:
}
ByteBuffer buffer = planes[i].getBuffer();
int rowStride = planes[i].getRowStride();
int pixelStride = planes[i].getPixelStride();
int shift = (i == 0) ? 0 : 1;
int w = width >> shift;
int h = height >> shift;
buffer.position(rowStride * (crop.top >> shift) + pixelStride * (crop.left >> shift));
for (int row = 0; row < h; row++) {
int length;
if (pixelStride == 1 && outputStride == 1) {
length = w;
buffer.get(mYuvBuffer, channelOffset, length);
channelOffset += length;
} else {
length = (w - 1) * pixelStride + 1;
buffer.get(rowData, 0, length);
for (int col = 0; col < w; col++) {
mYuvBuffer[channelOffset] = rowData[col * pixelStride];
channelOffset += outputStride;
}
}
if (row < h - 1) {
buffer.position(buffer.position() + rowStride - length);
}
}
}
}
@Override
protected int decodeType() {
return VIDEO;
}
@Override
public boolean handleMessage(@NonNull Message msg) {
switch (msg.what){
case MSG_VIDEO_OUTPUT:
try {
if (mTime == -1) {
mTime = System.currentTimeMillis();
}
Bundle bundle = msg.getData();
int index = bundle.getInt("index");
long ptsTime = bundle.getLong("time");
sleepRender(ptsTime,mTime);
// if (ptsTime + extractor.getSampleDiffTime() >= progress && progress >= ptsTime)
// {
// writeAudioFlag = true;
// }
// if (writeAudioFlag) {
mediaCodec.releaseOutputBuffer(index, false);
// }
} catch (Exception e) {
e.printStackTrace();
}
break;
default:break;
}
return super.handleMessage(msg);
}
/**
* 數(shù)據(jù)的時(shí)間戳對(duì)齊
**/
private long sleepRender(long ptsTimes, long startMs) {
/**
* 注意這里是以 0 為出事目標(biāo)的，info.presenttationTimes 的單位為微秒
* 這里用系統(tǒng)時(shí)間來模擬兩幀的時(shí)間差
*/
ptsTimes = ptsTimes / 1000;
long systemTimes = System.currentTimeMillis() - startMs;
long timeDifference = ptsTimes - systemTimes;
// 如果當(dāng)前幀比系統(tǒng)時(shí)間差快了，則延時(shí)以下
if (timeDifference > 0) {
try {
//todo 受系統(tǒng)影響，建議還是用視頻本身去告訴解碼器 pts 時(shí)間
Thread.sleep(timeDifference);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return timeDifference;
}
}
到了這里，關(guān)于Android音視頻編碼(2)的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！
