在英特尔开源WebRTC开发套件OWT Android版中加入音量检测
英特尔开源WebRTC协同通信开发套件(Intel® Collaboration Suite for WebRTC)。
WebRTC协同通信开发套件是一款端到端媒体服务器软件,客户端包含JavaScript,Android,iOS,Linux,Windows C++ SDK,服务端支持SFU和MCU模式,可实现实时音视频分析功能。
WebRTC协同通信开发套件开源后命名为Open WebRTC Toolkit(OWT),采用商业友好的Apache 2.0 License软件许可。
据Mark介绍,OWT保持了原有的Intel® Collaboration Suite for WebRTC的基本框架,并计划集成一系列新技术,如Scalable Video Technology(SVT)。此外,OWT开源社区还得到了阿里云、爱奇艺、Zealcomm、Huddl等企业支持。
OWT的代码已经在GitHub公开:https://github.com/open-webrtc-toolkit同时,Intel® Collaboration Suite for WebRTC项目webrtc.intel.com仍然会被保留,并作为英特尔官方的OWT分发版继续定期对外发布,当前最新版本是v4.2.1(2019年6月4日发布)。
在很多的应用场景中,我们需要在界面上显示自己说话的音量,和显示频道里其他用户的音量。在OWT提供的Android SDK中,有提供订阅的每一路流的的音量的接口。但在一些参于人多的场景中,我们需要订阅混合流,此时,我们就无法获取到每一个用户说话的音量。这里,就需要在发送端进行音量检测,并通过sendMessage接口,把音量数据发送到所有的接收者。
如果检测本地音量呢?
并无提供音量检测接口,如果要做音量检测,就需要获取到麦克风采集到的音频数据,并进行音量的计算。
首先,我们找到采集的入口,在WebRtcAudioRecord的AudioRecordThread
里,有调用一个audioSamplesReadyCallback.onWebRtcAudioRecordSamplesReady接口方法,这个方法默认是没有实现的,需要自己实现下这个方法
public class AudioProcessor implements JavaAudioDeviceModule.SamplesReadyCallback{ private int nCount = 0; private long nTotalVolume = 0; private long nSamples = 0; private double nInterval = 0.2;//计算间隔 ms IRtcEngineEventHandler mHandler; private ExecutorService executor = newSingleThreadExecutor(); @Override public void onWebRtcAudioRecordSamplesReady(JavaAudioDeviceModule.AudioSamples audioSamples) { //LogUtils.v("AudioProcessor", "onWebRtcAudioRecordSamplesReady" +" foramt:"+audioSamples.getAudioFormat()+" channel:"+audioSamples.getChannelCount()+" samplerate:"+audioSamples.getSampleRate()+ " length:"+audioSamples.getData().length); executor.execute(() -> { short[] data = bytesToShort(audioSamples.getData()); for (int i = 0; i<data.length; i++) { nTotalVolume += Math.abs(data[i]); } nSamples += data.length; if (nSamples >= audioSamples.getSampleRate() * nInterval ) { float fAverageVolume = (float) nTotalVolume/nSamples; double fVolume = 0; if (fAverageVolume >= 1) { fVolume = Math.log10(fAverageVolume)*20.0f; } if (mHandler != null) { mHandler.onLocalAudioVolumeIndication((int)fVolume); } //LogUtils.v("AudioProcessor", "10ms TotalVolume: " + nTotalVolume +" averageVolume:"+fAverageVolume + " FixedVolume: "+fVolume); nTotalVolume = 0; nSamples = 0; nCount = 0; } }); } public void setIRtcEngineEventHandler(IRtcEngineEventHandler handler) { mHandler = handler; } public static short[] bytesToShort(byte[] bytes) { if(bytes==null){ return null; } short[] shorts = new short[bytes.length/2]; ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts); return shorts; } }
新建一个AudioProcessor类,implements JavaAudioDeviceModule.SamplesReadyCallback
在onWebRtcAudioRecordSamplesReady回调方法里,实现检测,注意,不要在同一个线程里处理音频,不然,可能会导致采集延迟,从而使音频出现断掉的噪音。获取到的音频是byte型数据,每个sample占两字节,需转成short型,音量大小的计算可以用以下公式计算
fVolume = Math.log10(fAverageVolume)*20.0f;
考虑到AudioProcessor需要把采集的音量数据输出到应用层,建议在最外层初始化时创建对象,在ContextInitialization类里加入方法:
public ContextInitialization setSamplesReadyCallback(JavaAudioDeviceModule.SamplesReadyCallback samplesReadyCallback) { RCHECK(!initialized); PCFactoryProxy.samplesReadyCallback = samplesReadyCallback; return this; }
通过这个方法进行传入:
new ContextInitialization().setApplicationContext(context) .addIgnoreNetworkType(ContextInitialization.NetworkType.LOOPBACK) .setVideoHardwareAccelerationOptions( rootEglBase.getEglBaseContext(), rootEglBase.getEglBaseContext()) .setSamplesReadyCallback(mAudioProcessor) .initialize();
在
public void initialize() { RCHECK(!initialized); initialized = true; PCFactoryProxy.instance(); }
里,会调用到PeerConnectionFactory的instance()方法 static PeerConnectionFactory instance() { if (peerConnectionFactory == null) { PeerConnectionFactory.InitializationOptions initializationOptions = PeerConnectionFactory.InitializationOptions.builder(ContextInitialization.context) .setFieldTrials(fieldTrials) .createInitializationOptions(); PeerConnectionFactory.initialize(initializationOptions); PeerConnectionFactory.Options options = new PeerConnectionFactory.Options(); options.networkIgnoreMask = networkIgnoreMask; peerConnectionFactory = PeerConnectionFactory.builder() .setOptions(options) // .setAudioDeviceModule(adm == null ? new LegacyAudioDeviceModule() : adm) .setVideoEncoderFactory( encoderFactory == null ? new DefaultVideoEncoderFactory(localContext, true, true) : encoderFactory) .setVideoDecoderFactory( decoderFactory == null ? new DefaultVideoDecoderFactory(remoteContext) : decoderFactory) .setSamplesReadyCallback(samplesReadyCallback) .createPeerConnectionFactory(); } return peerConnectionFactory; } public PeerConnectionFactory createPeerConnectionFactory() { PeerConnectionFactory.checkInitializeHasBeenCalled(); if (this.audioDeviceModule == null) { this.audioDeviceModule = JavaAudioDeviceModule.builder(ContextUtils.getApplicationContext()) .setSamplesReadyCallback(this.samplesReadyCallback) .createAudioDeviceModule(); }
return PeerConnectionFactory.nativeCreatePeerConnectionFactory(ContextUtils.getApplicationContext(), this.options, this.audioDeviceModule.getNativeAudioDeviceModulePointer(), this.audioEncoderFactoryFactory.createNativeAudioEncoderFactory(), this.audioDecoderFactoryFactory.createNativeAudioDecoderFactory(), this.videoEncoderFactory, this.videoDecoderFactory, this.audioProcessingFactory == null ? 0L : this.audioProcessingFactory.createNative(), this.fecControllerFactoryFactory == null ? 0L : this.fecControllerFactoryFactory.createNative(), this.networkControllerFactoryFactory == null ? 0L : this.networkControllerFactoryFactory.createNativeNetworkControllerFactory(), this.networkStatePredictorFactoryFactory == null ? 0L : this.networkStatePredictorFactoryFactory.createNativeNetworkStatePredictorFactory(), this.mediaTransportFactoryFactory == null ? 0L : this.mediaTransportFactoryFactory.createNativeMediaTransportFactory(), this.neteqFactoryFactory == null ? 0L : this.neteqFactoryFactory.createNativeNetEqFactory()); }
这样,在JavaAudioDeviceModule的createAudioDeviceModule
方法里,在new WebRtcAudioRecord的时候,就可以把samplesReadyCallback传进去了。
这时,采集音频的时候,就可以把音频数据回调一份到Audioprocessor里进行处理了。
在这个方法里,也可以加入对音频数据处理的功能,参考前一篇视频处理的文章,加入AudioSink的功能,把处理后的音频回写到WebRtcAudioRecord。