Konrad Piwowar Posted March 3, 2021 Share Posted March 3, 2021 I'm getting thousands apon thousands of lines like this spammed into our antmedia error log (2.3.0 RC1 Feb 24th build date) [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized [sWR @ 0x7fe01c078700] Context has not been initialized [sWR @ 0x7fe01c08d6c0] Context has not been initialized [sWR @ 0x7fe01c03da40] Context has not been initialized Almost 5 gigs of this has been written in a few days. Any ideas? Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 3, 2021 Author Share Posted March 3, 2021 Other errors we are getting WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3]. Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true] Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener initializeSSL INFO: OpenSSL successfully initialized [OpenSSL 1.1.1 11 Sep 2018] Mar 03, 2021 9:36:41 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"] Mar 03, 2021 9:36:41 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"] Mar 03, 2021 9:36:41 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service [Tomcat] Mar 03, 2021 9:36:41 PM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/8.5.58 Mar 03, 2021 9:36:41 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment INFO: No global web.xml found Mar 03, 2021 9:36:41 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 03, 2021 9:36:42 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 03, 2021 9:36:42 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 03, 2021 9:36:42 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 03, 2021 9:36:42 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 03, 2021 9:36:43 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 03, 2021 9:36:43 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"] Mar 03, 2021 9:36:43 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"] (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. Antmedia has stopped responding 3 times today in the last few hours. We are doing WebRTC ingress and webrtc egress, no HLS or anythign else. Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 3, 2021 Author Share Posted March 3, 2021 The context has not been initialized error was spamming shortly before the daemon stopped responding. Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 3, 2021 Author Share Posted March 3, 2021 Sorry for the spam, trying to get all the info I can out for the devs to look at. The daemon has stopped responding quite a few times today it seems. Prior to today, nothing. -rw-r--r-- 1 antmedia antmedia 185760 Mar 3 16:51 hs_err_pid10035.log -rw-r--r-- 1 antmedia antmedia 222340 Mar 3 17:01 hs_err_pid10455.log -rw-r--r-- 1 antmedia antmedia 256260 Mar 3 18:09 hs_err_pid12515.log -rw-r--r-- 1 antmedia antmedia 311293 Mar 3 14:39 hs_err_pid1965.log -rw-r--r-- 1 antmedia antmedia 162788 Mar 3 14:39 hs_err_pid24133.log -rw-r--r-- 1 antmedia antmedia 161692 Mar 3 14:40 hs_err_pid24465.log -rw-r--r-- 1 antmedia antmedia 177154 Mar 3 14:41 hs_err_pid24772.log -rw-r--r-- 1 antmedia antmedia 196094 Mar 3 14:43 hs_err_pid25205.log -rw-r--r-- 1 antmedia antmedia 229247 Mar 3 18:09 hs_err_pid25336.log -rw-r--r-- 1 antmedia antmedia 192489 Mar 3 14:44 hs_err_pid25818.log -rw-r--r-- 1 antmedia antmedia 199314 Mar 3 18:10 hs_err_pid26138.log -rw-r--r-- 1 antmedia antmedia 149464 Mar 3 14:45 hs_err_pid26250.log -rw-r--r-- 1 antmedia antmedia 145194 Mar 3 14:47 hs_err_pid26550.log -rw-r--r-- 1 antmedia antmedia 158277 Mar 3 18:11 hs_err_pid26718.log -rw-r--r-- 1 antmedia antmedia 147102 Mar 3 14:48 hs_err_pid26811.log -rw-r--r-- 1 antmedia antmedia 239648 Mar 3 16:50 hs_err_pid27043.log -rw-r--r-- 1 antmedia antmedia 201374 Mar 3 18:13 hs_err_pid27050.log -rw-r--r-- 1 antmedia antmedia 188197 Mar 3 18:14 hs_err_pid27586.log -rw-r--r-- 1 antmedia antmedia 185486 Mar 3 18:15 hs_err_pid28057.log -rw-r--r-- 1 antmedia antmedia 197257 Mar 3 18:16 hs_err_pid28501.log -rw-r--r-- 1 antmedia antmedia 184783 Mar 3 18:17 hs_err_pid29006.log -rw-r--r-- 1 antmedia antmedia 177905 Mar 3 18:18 hs_err_pid29451.log -rw-r--r-- 1 antmedia antmedia 169537 Mar 3 18:20 hs_err_pid29777.log -rw-r--r-- 1 antmedia antmedia 169944 Mar 3 18:21 hs_err_pid30213.log -rw-r--r-- 1 antmedia antmedia 161182 Mar 3 18:22 hs_err_pid30495.log -rw-r--r-- 1 antmedia antmedia 172153 Mar 3 18:23 hs_err_pid30886.log -rw-r--r-- 1 antmedia antmedia 164505 Mar 3 18:24 hs_err_pid31280.log -rw-r--r-- 1 antmedia antmedia 148861 Mar 3 18:25 hs_err_pid31632.log -rw-r--r-- 1 antmedia antmedia 158891 Mar 3 18:26 hs_err_pid31877.log -rw-r--r-- 1 antmedia antmedia 156248 Mar 3 18:28 hs_err_pid32078.log -rw-r--r-- 1 antmedia antmedia 157205 Mar 3 18:29 hs_err_pid32315.log -rw-r--r-- 1 antmedia antmedia 250172 Mar 3 20:07 hs_err_pid32538.log -rw-r--r-- 1 antmedia antmedia 206044 Mar 3 20:07 hs_err_pid46973.log -rw-r--r-- 1 antmedia antmedia 226068 Mar 3 20:09 hs_err_pid47665.log -rw-r--r-- 1 antmedia antmedia 207825 Mar 3 20:10 hs_err_pid48972.log -rw-r--r-- 1 antmedia antmedia 212255 Mar 3 20:33 hs_err_pid5783.log -rw-r--r-- 1 antmedia antmedia 242304 Mar 3 20:25 hs_err_pid777.log -rw-r--r-- 1 antmedia antmedia 181275 Mar 3 20:33 hs_err_pid8738.log -rw-r--r-- 1 antmedia antmedia 157187 Mar 3 20:34 hs_err_pid9180.log -rw-r--r-- 1 antmedia antmedia 168797 Mar 3 20:35 hs_err_pid9485.log -rw-r--r-- 1 antmedia antmedia 190964 Mar 3 16:50 hs_err_pid9609.log -rw-r--r-- 1 antmedia antmedia 214222 Mar 3 20:39 hs_err_pid9765.log The "Context has not been initialized" error seems to happen around the same time as the daemon stops responding. It's impossible to stop it via "service antmedia stop" - we must kill the PID. The issue is very likely correlatated with someone trying to broadcast from Safari on an IPad (WebRTC Ingress) with the following user agent: 5.0 (iPad; CPU OS 12_4_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1 The server in question was crashing like crazy, and we moved that user to another server. The first server stopped crashing, the second server started. Link to comment Share on other sites More sharing options...
0 Selim Emre Posted March 3, 2021 Share Posted March 3, 2021 Hi Konrad, Thank you for reporting to us! We need to check the beginning of Ant Media Server logs. Could you please send us a full of logs? You can find ant-media-server.log file under the /usr/local/antmedia/log folder. Also, please send hs_err_pid10035.log file. Are you using Adaptive bitrate? Looking forward to hearing from you soon. Best Regards, Selim Link to comment Share on other sites More sharing options...
0 chandrashekar kola Posted March 4, 2021 Share Posted March 4, 2021 We also started noticing the same error for last few days. Do we know what can cause it. When ever we see this error, ant media server getting restarted. see below for the logs. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3]. Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true] Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener initializeSSL INFO: OpenSSL successfully initialized [OpenSSL 1.1.1 11 Sep 2018] Mar 03, 2021 7:53:49 PM org.apache.coyote.http11.AbstractHttp11Protocol configureUpgradeProtocol INFO: The ["http-apr-0.0.0.0-5080"] connector has been configured to support HTTP upgrade to [h2c] Mar 03, 2021 7:53:49 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"] Mar 03, 2021 7:53:49 PM org.apache.coyote.http11.AbstractHttp11Protocol configureUpgradeProtocol INFO: The ["https-openssl-apr-0.0.0.0-5443"] connector has been configured to support negotiation to [h2] via ALPN Mar 03, 2021 7:53:49 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"] Mar 03, 2021 7:53:49 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service [Tomcat] Mar 03, 2021 7:53:49 PM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/8.5.58 Mar 03, 2021 7:53:49 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment INFO: No global web.xml found Mar 03, 2021 7:53:49 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 03, 2021 7:53:50 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 03, 2021 7:53:50 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 03, 2021 7:53:51 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"] Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 4, 2021 Author Share Posted March 4, 2021 Selim, i've sent you the log file and a pid. We are still waiting for the user to reply with the exact device model they are using but it seems to be an old as hell iPad - the resolution we locked was 768x1024 so it's probably circa 2013-2014 at best. One of our staff noted we actually had an influx of crashes before as well when someone tried to stream from an old iPhone. It seems that these users are able to fairly consistently cause Antmedia daemon to stop responding just by streaming to it. We have for now disabled this users account as it was killing our production environment, and disabled streaming from iOS prior to v13 to prevent old devices from using the system. Staff member tried streaming through 2.3.0 RC1 with his IPhone XR (iOS 14) and the log is also flooding with these errors - albeit it didn't kill the daemon, so it seems to be recreatable by using Safari on an iPhone/iPad as WebRTC Ingress. Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 4, 2021 Author Share Posted March 4, 2021 Sorry missed the Adaptive streaming question - Yes, we are using adaptive. Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 4, 2021 Author Share Posted March 4, 2021 The user that was crashing our systems by streaming was using iPad A1475 which seems to be a first generation ipad air. Link to comment Share on other sites More sharing options...
0 Selim Emre Posted March 4, 2021 Share Posted March 4, 2021 Hi All, Thanks for the details. I'm checking this issue in our test environment. Best Regards, Selim Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 7, 2021 Author Share Posted March 7, 2021 And progress on this? We're still getting constant crashes, and it seems to trace back to iOS Safari WebRTC Ingress Link to comment Share on other sites More sharing options...
0 Selim Emre Posted March 8, 2021 Share Posted March 8, 2021 Hi Konrad, Sorry for the delay. I checked your issue. As I remember we did some improvements for this case in v2.3(after the RC version). Could you please try latest version? Best Regards, Selim Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 8, 2021 Author Share Posted March 8, 2021 The latest snapshot available for download is from March 2nd, prior to me reporting this. Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 8, 2021 Author Share Posted March 8, 2021 This issue is not resolved in the March 2nd build. We are also having substantial issues with audio not working for many broadcasters or cutting out. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. java.lang.NullPointerException at io.antmedia.enterprise.adaptive.AVFramePool.getAVFrame(AVFramePool.java:25) at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.onOpusPacket(WebRTCEncoderAdaptor.java:1007) at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.lambda$getAudioDecoderFactoryFactory$0(WebRTCEncoderAdaptor.java:960) at org.webrtc.BuiltinAudioDecoderFactoryFactory.onAudioPacket(BuiltinAudioDecoderFactoryFactory.java:41) Exception in thread "network_thread - 27933" # Fatal error in: gen/sdk/android/generated_builtin_audio_codes_jni/../../../../../../sdk/android/src/jni/jni_generator_helper.h, line 85 # last system error: 0 # Check failed: !env->ExceptionCheck() # WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3]. Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true]. Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true] Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener initializeSSL INFO: OpenSSL successfully initialized [OpenSSL 1.1.1 11 Sep 2018] Mar 08, 2021 12:48:16 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"] Mar 08, 2021 12:48:16 PM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"] Mar 08, 2021 12:48:16 PM org.apache.catalina.core.StandardService startInternal INFO: Starting service [Tomcat] Mar 08, 2021 12:48:16 PM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/8.5.58 Mar 08, 2021 12:48:16 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment INFO: No global web.xml found Mar 08, 2021 12:48:16 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 08, 2021 12:48:17 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 08, 2021 12:48:18 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 08, 2021 12:48:18 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 08, 2021 12:48:18 PM org.apache.catalina.core.ApplicationContext log INFO: No Spring WebApplicationInitializer types detected on classpath Mar 08, 2021 12:48:18 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize SEVERE: Spring context lookup failed, skipping spring component provider initialization. Mar 08, 2021 12:48:18 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"] Mar 08, 2021 12:48:18 PM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"] (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. (rtp_sender.cc:593): Tried to get DTMF sender from video sender. Link to comment Share on other sites More sharing options...
0 Selim Emre Posted March 8, 2021 Share Posted March 8, 2021 Hi Konrad, Thanks for the details. I will discuss this issue with our technical team and make an effort to get it resolved quickly. Best Regards, Selim Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 9, 2021 Author Share Posted March 9, 2021 Thanks - some more logs from times it crashed/stopped responding, or marked SEVERE: java.lang.NullPointerException at io.antmedia.enterprise.adaptive.AVFramePool.getAVFrame(AVFramePool.java:25) at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.onOpusPacket(WebRTCEncoderAdaptor.java:1007) at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.lambda$getAudioDecoderFactoryFactory$0(WebRTCEncoderAdaptor.java:960) at org.webrtc.BuiltinAudioDecoderFactoryFactory.onAudioPacket(BuiltinAudioDecoderFactoryFactory.java:41) Exception in thread "network_thread - 27933" # # Fatal error in: gen/sdk/android/generated_builtin_audio_codecs_jni/../../../../../../sdk/android/src/jni/jni_generator_helper.h, line 85 # last system error: 0 # Check failed: !env->ExceptionCheck() # WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release ----------------- Tons of this being spammed in the logs (thousands of entries), after we tried switching to Mono yesterday to try see if that resolves it. [opus @ 0x7f7a7012a1c0] Error decoding a SILK frame. [opus @ 0x7f7a7012a1c0] Error decoding an Opus frame. [opus @ 0x7f7a7012a1c0] Error decoding a SILK frame. [opus @ 0x7f7a7012a1c0] Error decoding an Opus frame. Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 9, 2021 Author Share Posted March 9, 2021 Another log from another unresponsiveness/crash today (2.3.0 March 2nd build) Mar 09, 2021 6:07:18 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process SEVERE: Error reading request, ignored java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [140,548,515,160,224] with wrapper [org.apache.tomcat.util.net.AprEndpoint$A$ at javax.websocket.CloseReason.<init>(CloseReason.java:70) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Mar 09, 2021 6:07:18 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process SEVERE: Error reading request, ignored java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [140,548,513,878,176] with wrapper [org.apache.tomcat.util.net.AprEndpoint$A$ at javax.websocket.CloseReason.<init>(CloseReason.java:70) at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153) at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818) at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 9, 2021 Author Share Posted March 9, 2021 And just to be clear as to what we consider unresponsive - the criteria for our script that catches and logs these issues is HTTP accessibility to Antmedia, so we are testing http://localhost:5080 and it takes longer then 5 seconds to respond we kill the pid and restart it. During timeouts like this, we are unable to do 'service antmedia stop' as it just hangs there. Link to comment Share on other sites More sharing options...
0 Selim Emre Posted March 9, 2021 Share Posted March 9, 2021 Hi Konrad, Thanks for the great feedback! Another log from another unresponsiveness/crash today (2.3.0 March 2nd build) Is there any reproduce scenario for this issue? Reproduce scenario is important for the understand real cause. Best Regards, Selim Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted March 9, 2021 Author Share Posted March 9, 2021 Unfortunately we haven't been able to identify anything in particular. The site just has performers streaming using WebRTC ingress and viewers using WebRTC Egress. It's random. Sometimes will happen 5x a day sometimes won't happen for 2 weeks. Sometimes there's 10 performers on a server and it's fine, sometimes with 1 it starts locking up. The only crash we were able to identify was caused by Safari WebRTC Ingress on a first gen Ipad Air but even when we blocked iOS < v13 we still have the daemon becoming unresponsive. The rest are using various devices, mostly chrome on pc, sometimes macs, sometimes android tablets or phones. Everything is done via browser. This has been going effectively since we started using the software a year ago and the only way we can keep everything going is by running our timeout script on a cron every 2 minutes checking the status, killing the pid and starting the daemon again. We've had alot of problems overall. Egress tends to be choppy for the first 15-20 seconds and then clears up - sometimes remains choppy but in some cases it is performer having terrible internet, 10 year old laptop or they are using wifi and apparently live in a Faraday cage. We suspected it might have to do with audio so we changed to Mono, which seemed to fix an issue with audio vanishing after a while but not the unresponsiveness. There's sometimes not even anything in the logs - it just becomes unresponsive and that's that. Most servers are 32 thread and 128-192gb of ram. We aren't hitting any resource limits. The amounts of viewers might be 50-100 on a server during peak. Link to comment Share on other sites More sharing options...
0 Selim Emre Posted March 9, 2021 Share Posted March 9, 2021 Hi Konrad, Thanks for all details. I will discuss this issue with our tech team. Best Regards, Selim Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted April 20, 2021 Author Share Posted April 20, 2021 Has any progress on this been made? Every time we crash we get this in the logs. Our error logs are several gigabytes in size... WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1 WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release I found this, maybe will help. https://github.com/spring-projects/spring-framework/issues/22674 Link to comment Share on other sites More sharing options...
0 Selim Emre Posted April 21, 2021 Share Posted April 21, 2021 Hi Konrad, Have a good day. Every time we crash we get this in the logs. Our error logs are several gigabytes in size... I'm sorry for the late response. Actually, I discussed this issue with my colleagues, but we couldn't reproduce this issue. Is there any specific reproduce scenario for this issue? Thank you for reporting this warning log issue. I added an issue for you -> https://github.com/ant-media/Ant-Media-Server/issues/3184 Best Regards, Selim Link to comment Share on other sites More sharing options...
0 Konrad Piwowar Posted April 21, 2021 Author Share Posted April 21, 2021 We are still experimenting with this. Thus far, we have seen one person be fairly consistently able to crash teh daemon by broadcasting from a very old macbook with Chrome 67. Not every time, but often. We have also seen this happen from some phones but also not always. Seems to be related to older apple devices we think. Right now we are playing wtih resolutions to see if this is maybe causing issue - basically going off 'what are we doing differently then everyone else' Link to comment Share on other sites More sharing options...
0 Selim Emre Posted April 21, 2021 Share Posted April 21, 2021 Hi Konrad, I also reported this issue in our issue tracking system -> https://github.com/ant-media/Ant-Media-Server/issues/3186 We will schedule to fix this issue. Best Regards, Selim Link to comment Share on other sites More sharing options...
Question
Konrad Piwowar
I'm getting thousands apon thousands of lines like this spammed into our antmedia error log (2.3.0 RC1 Feb 24th build date)
Link to comment
Share on other sites
Top Posters For This Question
16
8
1
Popular Days
Mar 9
6
Mar 3
5
Mar 4
5
Mar 8
4
Top Posters For This Question
Konrad Piwowar 16 posts
Selim Emre 8 posts
chandrashekar kola 1 post
Popular Days
Mar 9 2021
6 posts
Mar 3 2021
5 posts
Mar 4 2021
5 posts
Mar 8 2021
4 posts
24 answers to this question
Recommended Posts