Publish stopped with error cannot write audio frame to muxer

[chandrashekar kola]

Hello,

We are seeing publish being stopped randomly and all the viewers get disconnected from the stream. during the error we see below error and i also see bunch of Threads blocked errors. Let me know.

2021-02-25 19:01:48,071 [network_thread - 6519] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - onIceConnectionChange DISCONNECTED for streamId: the-montmartre-show-1 hash:1615060324

2021-02-25 19:01:48,071 [network_thread - 6519] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - entering stop() for the-montmartre-show-1

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - running stop operations in executor for stream: the-montmartre-show-1 Received audio frame: 209420  Processed audio frame: 209419Received video frame: 54586  Processed video frame: 54586

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Total video encode time 249868ms, average video encode time 4ms for stream: the-montmartre-show-1

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Total audio encode time 270102ms, average audio encode time 1ms for stream: the-montmartre-show-1

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Publish Stats StreamId: the-montmartre-show-1 TransferedByte: 124242919 Stream Duration: 2094197 Bitrate: 59 Source IP: 77.204.105.99 User-Agent: N/A

2021-02-25 19:01:48,075 [vert.x-eventloop-thread-8] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Encode video runner is returning because encoder adaptor is stopped for the-montmartre-show-1

2021-02-25 19:01:48,076 [vert.x-eventloop-thread-9] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Encode audio runner is returning because encoder adaptor is stopped for the-montmartre-show-1

2021-02-25 19:01:48,081 [vert.x-worker-thread-15] INFO  i.a.e.adaptive.StreamAdaptor - Recording set to false in streamadaptor: 2125715136 for stream: the-montmartre-show-1 stopping scheduler job id: 1970

2021-02-25 19:01:48,081 [vert.x-worker-thread-0] INFO  i.a.e.adaptive.StreamAdaptor - Recording set to false in streamadaptor: 426333455 for stream: the-montmartre-show-1 stopping scheduler job id: 1971

2021-02-25 19:01:48,081 [vert.x-worker-thread-0] INFO  i.a.e.adaptive.StreamAdaptor - Flushing video encoder:png for stream adaptor:426333455 stream id:the-montmartre-show-1

2021-02-25 19:01:48,081 [vert.x-worker-thread-15] INFO  io.antmedia.muxer.RecordMuxer - cannot write audio frame to muxer(the-montmartre-show-1-2021-02-25_18-26-53_720p.mp4). Error is Invalid argument^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^

2021-02-25 19:01:48,071 [network_thread - 6519] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - onIceConnectionChange DISCONNECTED for streamId: the-montmartre-show-1 hash:1615060324

2021-02-25 19:01:48,071 [network_thread - 6519] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - entering stop() for the-montmartre-show-1

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - running stop operations in executor for stream: the-montmartre-show-1 Received audio frame: 209420  Processed audio frame: 209419Received video frame: 54586  Processed video frame: 54586

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Total video encode time 249868ms, average video encode time 4ms for stream: the-montmartre-show-1

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Total audio encode time 270102ms, average audio encode time 1ms for stream: the-montmartre-show-1

2021-02-25 19:01:48,071 [vert.x-eventloop-thread-6] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Publish Stats StreamId: the-montmartre-show-1 TransferedByte: 124242919 Stream Duration: 2094197 Bitrate: 59 Source IP: 77.204.105.99 User-Agent: N/A

2021-02-25 19:01:48,075 [vert.x-eventloop-thread-8] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Encode video runner is returning because encoder adaptor is stopped for the-montmartre-show-1

2021-02-25 19:01:48,076 [vert.x-eventloop-thread-9] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Encode audio runner is returning because encoder adaptor is stopped for the-montmartre-show-1

2021-02-25 19:01:48,081 [vert.x-worker-thread-15] INFO  i.a.e.adaptive.StreamAdaptor - Recording set to false in streamadaptor: 2125715136 for stream: the-montmartre-show-1 stopping scheduler job id: 1970

2021-02-25 19:01:48,081 [vert.x-worker-thread-0] INFO  i.a.e.adaptive.StreamAdaptor - Recording set to false in streamadaptor: 426333455 for stream: the-montmartre-show-1 stopping scheduler job id: 1971

2021-02-25 19:01:48,081 [vert.x-worker-thread-0] INFO  i.a.e.adaptive.StreamAdaptor - Flushing video encoder:png for stream adaptor:426333455 stream id:the-montmartre-show-1

 

2021-02-25 19:01:48,081 [vert.x-worker-thread-15] INFO  io.antmedia.muxer.RecordMuxer - cannot write audio frame to muxer(the-montmartre-show-1-2021-02-25_18-26-53_720p.mp4). Error is Invalid argument^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^

[Cem K1l1]

Hello,

Can you give me the details of your instance type(cpu,memory etc), ant media server version and publishing methods? Also do you have adaptive bitrate enabled?

Does it starts after viewer number reach to some point? Or can it happen even without any viewers?

[chandrashekar kola]

We are using g4dn.2xlarge type aws instance. it has 8 vCPU and 32 GB memory. 

ant media version: 2.2.1

Publish method: webrtc

Adaptive bit rate : enabled

I have noticed when the viewers count is between 80 and 100 and CPU is between 30% and 40%. But when i ran the load test using the script from ant media, i was able to reach to 500 viewers. 

Let me know. Thanks

[Cem K1l1]

Hi,

Ant media v2.3 will be available soon so I suggest you upgrade your version if it won't be too much trouble for you. Many of the memory and CPU problems are optimized. 

Also, how many broadcasts are you publishing and how many adaptive bit rate configurations have you enabled? This may cause extra load on your CPU. Can you it for us please?

[chandrashekar kola]

Sure i will upgrade when it is available on AWS. But we are actually having lot of issues in production due to this. 

The max broadcasts at a time are around 3, if not 2.  See other info below. Let me know if you need any other info.

WebRTC Codec Support : H264

Frame rate: 25

Resolution: 720p

Video Bitrate: 1500

Audio bitrate: 128

[Cem K1l1]

Hello,

Did you update to v2.3. Can you give me any updates about this issue?

 

[chandrashekar kola]

Yes we have upgraded it and started seeing similar issue where the ant media server crashes. Based on the support ticket they were asking us to install 2.3.1 snapshot (the bug we got into has been fixed).

 

 

[Cem K1l1]

Hello, 

Since it's been a long time, and you have updated the server, can you tell the exact problem that you are facing now with the new version in detail?

 

[chandrashekar kola]

The problem is that randomly users get disconnected from the streaming. It is not just few users, almost all of the users, as if the antmedia jvm crashed. Here is what we see in the logs, when it happens. Any help appreciated. We are stuck with this issue for few weeks now. We were on 2.2 when we saw this issue initially, upgraded 2.3.0. but no luck.

 

2021-04-09 15:09:32,168 [vert.x-eventloop-thread-0] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Publish Stats StreamId: ******* TransferedByte: 65825280 Stream Duration: 667811 Bitrate: 98 Source IP: 92.184.97.97 User-Agent: N/A

2021-04-09 15:09:32,171 [vert.x-worker-thread-4] INFO  i.a.e.adaptive.StreamAdaptor - Recording set to false in streamadaptor: 1051736214 for stream: *******   stopping scheduler job id: 11

2021-04-09 15:09:32,171 [vert.x-worker-thread-4] INFO  io.antmedia.muxer.RecordMuxer - cannot write audio frame to muxer(******* ). Error is Invalid argument^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

2021-04-09 15:09:32,171 [vert.x-worker-thread-4] INFO  io.antmedia.muxer.HLSMuxer - cannot write frame(not video) to muxer. Error is Invalid argument^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

2021-04-09 15:09:32,171 [vert.x-worker-thread-4] WARN  org.webrtc.audio.WebRtcAudioRecord - Discarding audio packet because audio packet size(5237) is bigger than buffer capacity3200 and limit 3200

2021-04-09 15:09:32,171 [vert.x-worker-thread-4] WARN  org.webrtc.audio.WebRtcAudioRecord - Discarding audio packet because audio packet size(5237) is bigger than buffer capacity3200 and limit 3200

2021-04-09 15:09:32,171 [vert.x-worker-thread-4] WARN  org.webrtc.audio.WebRtcAudioRecord - Discarding audio packet because audio packet size(5237) is bigger than buffer capacity3200 and limit 3200

2021-04-09 15:09:32,172 [vert.x-worker-thread-4] WARN  org.webrtc.audio.WebRtcAudioRecord - Discarding audio packet because audio packet size(5237) is bigger than buffer capacity3200 and limit 3200

2021-04-09 15:09:32,172 [vert.x-worker-thread-4] INFO  io.antmedia.muxer.RecordMuxer - cannot write audio frame to muxer(*******). Error is Invalid argument^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

2021-04-09 15:09:32,172 [vert.x-worker-thread-4] INFO  io.antmedia.muxer.HLSMuxer - cannot write frame(not video) to muxer. Error is Invalid argument^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

2021-04-09 15:09:32,172 [vert.x-worker-thread-4] WARN  org.webrtc.audio.WebRtcAudioRecord - Discarding audio packet because audio packet size(6748) is bigger than buffer capacity3200 and limit 3200

2021-04-09 15:09:32,172 [vert.x-worker-thread-4] WARN  org.webrtc.audio.WebRtcAudioRecord - Discarding audio packet because audio packet size(6748) is bigger than buffer capacity3200 and limit 3200

2021-04-09 15:09:32,173 [vert.x-worker-thread-4] ERROR i.a.e.adaptive.video.H264Encoder - Cannot encode video frame for stream index 0 error is End of file^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ for stream:*******

2021-04-09 15:09:32,173 [vert.x-worker-thread-4] WARN  i.a.e.adaptive.video.H264Encoder - There is no video output at this time for stream index 0 for stream:*******

2021-04-09 15:09:32,173 [vert.x-worker-thread-4] INFO  i.a.e.adaptive.StreamAdaptor - Flushing video encoder:h264_nvenc for stream adaptor:1051736214 stream id:*******

2021-04-09 15:09:32,174 [vert.x-worker-thread-4] INFO  i.a.e.adaptive.StreamAdaptor - Flushing audio encoder:libopus for stream adaptor:1051736214 stream id:*******

2021-04-09 15:09:32,175 [vert.x-worker-thread-4] INFO  i.a.e.adaptive.StreamAdaptor - Flushing audio encoder:aac for stream adaptor:1051736214 stream id:*******

2021-04-09 15:09:32,175 [vert.x-worker-thread-4] INFO  io.antmedia.muxer.RecordMuxer - Record Muxer writing trailer for stream: *******

2021-04-09 15:09:32,176 [vert.x-eventloop-thread-2] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Encode video runner is returning because encoder adaptor is stopped for *******

2021-04-09 15:09:32,177 [vert.x-eventloop-thread-3] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - Encode audio runner is returning because encoder adaptor is stopped for *******

2021-04-09 15:09:32,178 [network_thread - 2283] INFO  org.webrtc.Logging - Tag:WebRtcAudioTrackExternal - Message:GetPlayoutUnderrunCount