[Urgent] Streaming ending abruptly

[Chai]

Hi,

Our servers are consistently running into this issue which is causing major bottlenecks in our workflows. 

Any ideas how we can resolve this issue?

May 11, 2021 8:33:26 PM io.vertx.core.impl.BlockedThreadChecker

WARNING: Thread Thread[vert.x-eventloop-thread-31,5,main]=Thread[vert.x-eventloop-thread-31,5,main] has been blocked for 6498801 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

at io.antmedia.enterprise.webrtc.WebRTCMuxer.unRegisterWebRTCClient(WebRTCMuxer.java:246)

at io.antmedia.enterprise.webrtc.WebRTCClient.stopPartial(WebRTCClient.java:1094)

at io.antmedia.enterprise.webrtc.WebRTCClient.lambda$stop$17(WebRTCClient.java:1137)

at io.antmedia.enterprise.webrtc.WebRTCClient$$Lambda$545/0x00000008407b3840.run(Unknown Source)

at io.antmedia.enterprise.webrtc.WebRTCClient.lambda$executeOnSignallingThread$12(WebRTCClient.java:965)

at io.antmedia.enterprise.webrtc.WebRTCClient$$Lambda$510/0x0000000840796c40.handle(Unknown Source)

at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)

at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)

at io.vertx.core.impl.EventLoopContext$$Lambda$485/0x000000084072f040.run(Unknown Source)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

at java.base@ 11.0.10/java.lang.Thread.run(Thread.java:834)

[Tahir]

Hi,

Thread deadlock problem is resolved now, it will be deployed in the next coming version. Thank you for your patience.

Cheers

[Chai]

Thanks, Tahir for the prompt response.

Is there something we can fix on our AWS instances interim while we wait for the next coming version? 

Also - any insights on the root cause analysis of how this issue got introduced?

[Tahir]

Hi,

 

I fear it is not possible since it is related with the source code of the server. If you are using the license from Ant Media on AWS I can send you a SNAPSHOT maybe?

The issue was, in summary a synchronization problem: https://docs.oracle.com/javase/tutorial/essential/concurrency/syncmeth.html

Cheers

 

[Chai]

Yes,

Can you send us the snapshot?

Also, we are getting publish timeout errors. How can we solve this?

TCEncoderAdaptor - Encode video runner is returning because encoder adaptor is stopped for explore-the-australian-wilderness-1

2021-05-15 21:29:26,659 [vert.x-eventloop-thread-16] INFO  i.a.e.adaptive.WebRTCEncoderAdaptor - leaving stop() for explore-the-australian-wilderness-1

2021-05-15 21:29:26,659 [vert.x-eventloop-thread-16] INFO  i.a.AntMediaApplicationAdapter - Closing broadcast stream id: explore-the-australian-wilderness-1

2021-05-15 21:29:26,668 [vert.x-eventloop-thread-16] INFO  i.a.AntMediaApplicationAdapter - Setting timer to call live stream ended hook for stream:explore-the-australian-wilderness-1

2021-05-15 21:29:26,708 [vert.x-eventloop-thread-27] INFO  i.a.AntMediaApplicationAdapter - POST Response Status:: 301

2021-05-15 21:29:30,810 [https-openssl-nio2-0.0.0.0-5443-exec-7] INFO  i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:32,545 [https-openssl-nio2-0.0.0.0-5443-exec-3] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException: 

2021-05-15 21:29:36,572 [https-openssl-nio2-0.0.0.0-5443-exec-2] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException: 

2021-05-15 21:29:36,781 [https-openssl-nio2-0.0.0.0-5443-exec-9] INFO  i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:39,663 [https-openssl-nio2-0.0.0.0-5443-exec-4] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException: 

2021-05-15 21:29:39,746 [https-openssl-nio2-0.0.0.0-5443-exec-7] INFO  i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:44,039 [https-openssl-nio2-0.0.0.0-5443-exec-2] INFO  i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:44,679 [https-openssl-nio2-0.0.0.0-5443-exec-6] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException: 

2021-05-15 21:29:54,355 [vert.x-eventloop-thread-0] INFO  i.antmedia.statistic.StatsCollector - System cpu load:0 process cpu load:0 available memory: 63138384 KB used memory(RSS): 984432 KB

2021-05-15 21:29:54,356 [vert.x-eventloop-thread-0] INFO  i.antmedia.statistic.StatsCollector - Vertx worker queue size:0 WebRTCVertx worker queue size:0

2021-05-15 21:29:55,474 [https-openssl-nio2-0.0.0.0-5443-exec-10] INFO  i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:34:50,938 [https-openssl-nio2-0.0.0.0-5443-exec-1] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException: 

2021-05-15 21:34:51,704 [https-openssl-nio2-0.0.0.0-5443-exec-1] INFO  i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:34:54,355 [vert.x-eventloop-thread-0] INFO  i.antmedia.statistic.StatsCollector - System cpu load:0 process cpu load:0 available memory: 63132260 KB used memory(RSS): 989796 KB

2021-05-15 21:34:54,355 [vert.x-eventloop-thread-0] INFO  i.antmedia.statistic.StatsCollector - Vertx worker queue size:0 WebRTCVertx worker queue size:0

2021-05-15 21:34:55,068 [https-openssl-nio2-0.0.0.0-5443-exec-4] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException: 

[Tahir]

Please send an e-mail to support@antmedia.io for the snapshot.

We need to check your ICE connection process, can you elaborate your way of sending the broadcast? Also, you should check your firewall, for most of the time UDP ports are getting blocked for webRTC and people get publish time out errors. So please make sure that UDP ports between 5000-65000 are open for server and there are no blockage at the broadcaster.

 

[Chai]

Tahir,

Thanks for the reply. We will check our UDP ports. We don’t have a firewall, but our web application is on Heroku and Ant Instance is on AWS.  

Our process outline:

Our web application is hosted on Heroku. Our Ant Media servers are on AWS. A webpage with integrated WebRTC code will publish the stream which publishes media files to Ant Instance on AWS. 

The viewer pages(on web application) hosted on Heroku will “fetch” for the streams and play them. 

- Tahir 

--

Chaitanya Malla