one video stream with multiple audio tracks ?

[Alex A]

Hi,

Simple question: is it possible to have a single video stream which contains multiple (7x different) audio tracks, and on the client-side be able to select which audio tracks in the video stream should be played ?

This is because we want to make a very specific mergestream which contains 7 different audio tracks, and on the client side we want to play all the audio EXCEPT for individual audio tracks which the client can 'mute'.

So imagine a mergestream feed (single stream) which the viewers can MUTE the audio of individual participants within that stream.

So to do this, we imagine we would like 7x audio tracks inside the video stream, and be able to play/mute each audio track individually.

----

We could of course setup 7 independent streams, one for each audio track, but we are trying to simplify this and contain it within one stream.

Is this possible ?

Thanks!

Alex.

[Tahir]

Hi Alex,

This feature is in development right now in the server side. What I did so far for the feature is in these issues. It requires a bit of architectural change to support multiple tracks in the same stream so we are working on that. Also, there might be a solution in the application level for such task, stream_merger.js as we talked has a mute flag also. If you publish to a conference room with multiple audio tracks you can merge them on client side and mute the ones you like with mute flag, however, in this case you need to develop a system that the watcher sends commands to the merger client for muting and unmuting. I hope I could clearly explain this, if you want to go that direction I can help more with the mute flag.

https://github.com/ant-media/Ant-Media-Server/issues/2947

https://github.com/ant-media/Ant-Media-Server/issues/3028

Cheers

 

[Alex A]

Hi Tahir,

Thanks for your help - yes i understood what you meant. 

The problem is i want each viewer to independently mute some audio feeds in the merged stream...  for example there are 6 people in the boradcast but 50x viewers, and some viewers may want to mute person1, but others do not.  So sending a mute command back to the merger for muting tasks will not help.

What I did for now is to broadcast a merged stream (singular) with 7x audio streams separately... the timing looks okay, they do not seem to become 'out of sync' with the video - so this appears to work okay.

Thanks!

Alex.

 

[Tahir]

Hi,

Yep, I got your request, like I said it is in development right now, actually some part is done. You will be able to send websocket messages from client to the server about which audio tracks you want to disable, it is done for video but we needed to change low level architecture for audio, check out the PR: https://github.com/ant-media/StreamApp/pull/168/files

I think it will be there for 2.3.4 version, it will not catch 2.3.3 .

Cheers

 

[Peter Zanetti]

What is the application here for video as it is being implemented? Multiple video tracks in a single stream?

 

 

[Tahir]

Correct, it is multiple video tracks and audio tracks in a single stream.

[Yan]

Hi Everyone,

I'm jumped on this thread, but do we have any update about this kind of features ?
The filter api is not enough to handle real map especially when we need to deliver in HLS at the end.

Thank you for your kind follow up.

[Mohit]

On 3/5/2023 at 3:52 PM, Yan said:

Hi Everyone,

I'm jumped on this thread, but do we have any update about this kind of features ?
The filter api is not enough to handle real map especially when we need to deliver in HLS at the end.

Thank you for your kind follow up.

Hi Yan,

How are you?

Multitrack publish and play is readily available on Ant Media Server. Also we have multitrack support for the native Android and iOS SDKs.

You can refer to this blogpost for a quick introduction to multitrack feature.

Thank you.  

[Yan]

Hi Mohit,

Thanks for your kind reply. I see there is possibility yes. But unfortunately I didn't see how we can fix the HLS map creation in relation with this. 

If I take an exemple for 4 languages with 4 videos quality, I process it like this for VOD processing. So my map is generated accordingly with languages I have. 

I tried playing with ant to understand if we can do this kind of stuff, but cannot find my answer at this point, still searching 🙂 Oups, I'm searching a solution for livestreaming, not for VOD - to clarify.

Also, we produce live with OBS, using the 6 audio group (pid) in stereo with ffmpeg on record tabs. We push it in SRT.
Your experience will be welcome and appreciated.

Yan

 

-filter_complex:v [0:v]scale=1280:-2,fps=25[vout0] \

-map [vout0] \

-force_key_frames:v:0 expr:eq(mod(n,50),0) \

-x264opts:v:0 rc-lookahead=50:keyint=100:min-keyint=50:no-open-gop=1 \

-preset:v:0 veryfast \

-pix_fmt:v:0 yuv420p \

-c:v:0 libx264 \

-maxrate:v 4900k \

-bufsize:v 4M \

-b:v:0 4500k \

-filter_complex:v [0:v]scale=960:-2,fps=25[vout1] \

-map [vout1] \

-force_key_frames:v:1 expr:eq(mod(n,50),0) \

-x264opts:v:1 rc-lookahead=50:keyint=100:min-keyint=50:no-open-gop=1 \

-preset:v:1 veryfast \

-pix_fmt:v:1 yuv420p \

-c:v:1 libx264 \

-maxrate:v 2700k \

-bufsize:v 2M \

-b:v:1 2500k \

-filter_complex:v [0:v]scale=640:-2,fps=25[vout2] \

-map [vout2] \

-force_key_frames:v:2 expr:eq(mod(n,50),0) \

-x264opts:v:2 rc-lookahead=50:keyint=100:min-keyint=50:no-open-gop=1 \

-preset:v:2 veryfast \

-pix_fmt:v:2 yuv420p \

-c:v:2 libx264 \

-maxrate:v:2 1320k \

-bufsize:v:2 1M \

-b:v:1 1200k \

-filter_complex:v [0:v]scale=320:-2,fps=12[vout3] \

-map [vout3] \

-force_key_frames:v:3 expr:eq(mod(n,25),0) \

-x264opts:v:3 rc-lookahead=30:keyint=50:min-keyint=25:no-open-gop=1 \

-preset:v:3 veryfast \

-pix_fmt:v:3 yuv420p \

-c:v:3 libx264 \

-minrate:v:3 90k \

-maxrate:v:3 110k \

-bufsize:v:3 100k \

-b:v:3 100k \

-filter_complex:a [0:a:0]pan=mono|c0=c0[aout0];[0:a:1]pan=mono|c0=c0[aout1];[0:a:3]pan=mono|c0=c0[aout3];[0:a:5]pan=mono|c0=c0[aout5] \

-map [aout0] \

-map [aout1] \

-map [aout3] \

-map [aout5] \

-c:a aac \

-b:a 96k \

-var_stream_map "a:0,agroup:hls-4lang-fl-en-es-ru-4v,default:yes,language:Floor,name:Floor a:1,agroup:hls-4lang-fl-en-es-ru-4v,language:English,name:English a:3,agroup:hls-4lang-fl-en-es-ru-4v,language:Spanish,name:Spanish a:5,agroup:hls-4lang-fl-en-es-ru-4v,language:Russian,name:Russian v:0,agroup:hls-4lang-fl-en-es-ru-4v,name:1280 v:1,agroup:hls-4lang-fl-en-es-ru-4v,name:960 v:2,agroup:hls-4lang-fl-en-es-ru-4v,name:640 v:3,agroup:hls-4lang-fl-en-es-ru-4v,name:320" \

-hls_time 5 \

-flags +global_header+cgop \

-movflags +faststart \

-err_detect compliant \

-copyts \

-muxdelay 0 \

-hls_segment_type fmp4 \

-f hls \

-hls_list_size 0 \

-hls_playlist_type vod \

-hls_flags single_file+program_date_time+independent_segments+round_durations \

-hls_segment_filename #{out.dir}/#{out.name}_segment_%v.mp4 \

-master_pl_name #{out.name}-master.m3u8 \

#{out.dir}/#{out.name}_variant_%v#{out.suffix}

Edited March 6 by Yan

[robertosamayoa]

On 5/31/2021 at 4:09 PM, Alex A said:

What I did for now is to broadcast a merged stream (singular) with 7x audio streams separately... the timing looks okay, they do not seem to become 'out of sync' with the video - so this appears to work okay

Here you can control audio and video source from merge page ?

[Mohit]

On 3/6/2023 at 9:41 PM, Yan said:

Hi Mohit,

Thanks for your kind reply. I see there is possibility yes. But unfortunately I didn't see how we can fix the HLS map creation in relation with this. 

If I take an exemple for 4 languages with 4 videos quality, I process it like this for VOD processing. So my map is generated accordingly with languages I have. 

I tried playing with ant to understand if we can do this kind of stuff, but cannot find my answer at this point, still searching 🙂 Oups, I'm searching a solution for livestreaming, not for VOD - to clarify.

Also, we produce live with OBS, using the 6 audio group (pid) in stereo with ffmpeg on record tabs. We push it in SRT.
Your experience will be welcome and appreciated.

Yan

 

-filter_complex:v [0:v]scale=1280:-2,fps=25[vout0] \

-map [vout0] \

-force_key_frames:v:0 expr:eq(mod(n,50),0) \

-x264opts:v:0 rc-lookahead=50:keyint=100:min-keyint=50:no-open-gop=1 \

-preset:v:0 veryfast \

-pix_fmt:v:0 yuv420p \

-c:v:0 libx264 \

-maxrate:v 4900k \

-bufsize:v 4M \

-b:v:0 4500k \

-filter_complex:v [0:v]scale=960:-2,fps=25[vout1] \

-map [vout1] \

-force_key_frames:v:1 expr:eq(mod(n,50),0) \

-x264opts:v:1 rc-lookahead=50:keyint=100:min-keyint=50:no-open-gop=1 \

-preset:v:1 veryfast \

-pix_fmt:v:1 yuv420p \

-c:v:1 libx264 \

-maxrate:v 2700k \

-bufsize:v 2M \

-b:v:1 2500k \

-filter_complex:v [0:v]scale=640:-2,fps=25[vout2] \

-map [vout2] \

-force_key_frames:v:2 expr:eq(mod(n,50),0) \

-x264opts:v:2 rc-lookahead=50:keyint=100:min-keyint=50:no-open-gop=1 \

-preset:v:2 veryfast \

-pix_fmt:v:2 yuv420p \

-c:v:2 libx264 \

-maxrate:v:2 1320k \

-bufsize:v:2 1M \

-b:v:1 1200k \

-filter_complex:v [0:v]scale=320:-2,fps=12[vout3] \

-map [vout3] \

-force_key_frames:v:3 expr:eq(mod(n,25),0) \

-x264opts:v:3 rc-lookahead=30:keyint=50:min-keyint=25:no-open-gop=1 \

-preset:v:3 veryfast \

-pix_fmt:v:3 yuv420p \

-c:v:3 libx264 \

-minrate:v:3 90k \

-maxrate:v:3 110k \

-bufsize:v:3 100k \

-b:v:3 100k \

-filter_complex:a [0:a:0]pan=mono|c0=c0[aout0];[0:a:1]pan=mono|c0=c0[aout1];[0:a:3]pan=mono|c0=c0[aout3];[0:a:5]pan=mono|c0=c0[aout5] \

-map [aout0] \

-map [aout1] \

-map [aout3] \

-map [aout5] \

-c:a aac \

-b:a 96k \

-var_stream_map "a:0,agroup:hls-4lang-fl-en-es-ru-4v,default:yes,language:Floor,name:Floor a:1,agroup:hls-4lang-fl-en-es-ru-4v,language:English,name:English a:3,agroup:hls-4lang-fl-en-es-ru-4v,language:Spanish,name:Spanish a:5,agroup:hls-4lang-fl-en-es-ru-4v,language:Russian,name:Russian v:0,agroup:hls-4lang-fl-en-es-ru-4v,name:1280 v:1,agroup:hls-4lang-fl-en-es-ru-4v,name:960 v:2,agroup:hls-4lang-fl-en-es-ru-4v,name:640 v:3,agroup:hls-4lang-fl-en-es-ru-4v,name:320" \

-hls_time 5 \

-flags +global_header+cgop \

-movflags +faststart \

-err_detect compliant \

-copyts \

-muxdelay 0 \

-hls_segment_type fmp4 \

-f hls \

-hls_list_size 0 \

-hls_playlist_type vod \

-hls_flags single_file+program_date_time+independent_segments+round_durations \

-hls_segment_filename #{out.dir}/#{out.name}_segment_%v.mp4 \

-master_pl_name #{out.name}-master.m3u8 \

#{out.dir}/#{out.name}_variant_%v#{out.suffix}

Hi @Yan,

Thank you for the detailed explanation on this.

However to be honest, I'm not sure about this particular case with regards to HLS.

Let me request my senior colleague to help us with this.

Thank you 

[Yan]

7 minutes ago, Mohit said:

Hi @Yan,

Thank you for the detailed explanation on this.

However to be honest, I'm not sure about this particular case with regards to HLS.

Let me request my senior colleague to help us with this.

Thank you 

Hi @Mohit,

Thank you for your kind follow up. I send you a mail on the support with a more detailed description. In fact it's a standard HLS feature, there is just multiple method to implement it from the source stream to the m3u8. I copy you some code I'm using on our media manager to process the files for VoD on my last post. Just to show the point with audio channels in HLS: we have to be able to add multiple bitrate, multiple audio track and groups are set in an index.

AWS Medialive do this kind of thinks with an understandable display of the information, I copy you part of the screen. This can be done by ffmpeg also or other kind of media processor.

 I will wait for your kind follow up after your discussion with your team, again, thanks for your time on this discussion. 

Best,
Yan