Jump to content
  • 0

Context has not been initialized


Konrad Piwowar
 Share

Question

I'm getting thousands apon thousands of lines like this spammed into our antmedia error log (2.3.0 RC1 Feb 24th build date)

[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized
[sWR @ 0x7fe01c078700] Context has not been initialized
[sWR @ 0x7fe01c08d6c0] Context has not been initialized
[sWR @ 0x7fe01c03da40] Context has not been initialized

Almost 5 gigs of this has been written in a few days. 
Any ideas?
Link to comment
Share on other sites

  • Answers 24
  • Created
  • Last Reply

Top Posters For This Question

24 answers to this question

Recommended Posts

  • 0

Other errors we are getting

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3].
Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]
Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener initializeSSL
INFO: OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]
Mar 03, 2021 9:36:41 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"]
Mar 03, 2021 9:36:41 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]
Mar 03, 2021 9:36:41 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Mar 03, 2021 9:36:41 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/8.5.58
Mar 03, 2021 9:36:41 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Mar 03, 2021 9:36:41 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Mar 03, 2021 9:36:42 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize
SEVERE: Spring context lookup failed, skipping spring component provider initialization.
Mar 03, 2021 9:36:42 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Mar 03, 2021 9:36:42 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize
SEVERE: Spring context lookup failed, skipping spring component provider initialization.
Mar 03, 2021 9:36:42 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Mar 03, 2021 9:36:43 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize
SEVERE: Spring context lookup failed, skipping spring component provider initialization.
Mar 03, 2021 9:36:43 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"]
Mar 03, 2021 9:36:43 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

Antmedia has stopped responding 3 times today in the last few hours. We are doing WebRTC ingress and webrtc egress, no HLS or anythign else.

Link to comment
Share on other sites

  • 0

Sorry for the spam, trying to get all the info I can out for the devs to look at. The daemon has stopped responding quite a few times today it seems. Prior to today, nothing.

-rw-r--r-- 1 antmedia antmedia  185760 Mar  3 16:51 hs_err_pid10035.log
-rw-r--r-- 1 antmedia antmedia  222340 Mar  3 17:01 hs_err_pid10455.log
-rw-r--r-- 1 antmedia antmedia  256260 Mar  3 18:09 hs_err_pid12515.log
-rw-r--r-- 1 antmedia antmedia  311293 Mar  3 14:39 hs_err_pid1965.log
-rw-r--r-- 1 antmedia antmedia  162788 Mar  3 14:39 hs_err_pid24133.log
-rw-r--r-- 1 antmedia antmedia  161692 Mar  3 14:40 hs_err_pid24465.log
-rw-r--r-- 1 antmedia antmedia  177154 Mar  3 14:41 hs_err_pid24772.log
-rw-r--r-- 1 antmedia antmedia  196094 Mar  3 14:43 hs_err_pid25205.log
-rw-r--r-- 1 antmedia antmedia  229247 Mar  3 18:09 hs_err_pid25336.log
-rw-r--r-- 1 antmedia antmedia  192489 Mar  3 14:44 hs_err_pid25818.log
-rw-r--r-- 1 antmedia antmedia  199314 Mar  3 18:10 hs_err_pid26138.log
-rw-r--r-- 1 antmedia antmedia  149464 Mar  3 14:45 hs_err_pid26250.log
-rw-r--r-- 1 antmedia antmedia  145194 Mar  3 14:47 hs_err_pid26550.log
-rw-r--r-- 1 antmedia antmedia  158277 Mar  3 18:11 hs_err_pid26718.log
-rw-r--r-- 1 antmedia antmedia  147102 Mar  3 14:48 hs_err_pid26811.log
-rw-r--r-- 1 antmedia antmedia  239648 Mar  3 16:50 hs_err_pid27043.log
-rw-r--r-- 1 antmedia antmedia  201374 Mar  3 18:13 hs_err_pid27050.log
-rw-r--r-- 1 antmedia antmedia  188197 Mar  3 18:14 hs_err_pid27586.log
-rw-r--r-- 1 antmedia antmedia  185486 Mar  3 18:15 hs_err_pid28057.log
-rw-r--r-- 1 antmedia antmedia  197257 Mar  3 18:16 hs_err_pid28501.log
-rw-r--r-- 1 antmedia antmedia  184783 Mar  3 18:17 hs_err_pid29006.log
-rw-r--r-- 1 antmedia antmedia  177905 Mar  3 18:18 hs_err_pid29451.log
-rw-r--r-- 1 antmedia antmedia  169537 Mar  3 18:20 hs_err_pid29777.log
-rw-r--r-- 1 antmedia antmedia  169944 Mar  3 18:21 hs_err_pid30213.log
-rw-r--r-- 1 antmedia antmedia  161182 Mar  3 18:22 hs_err_pid30495.log
-rw-r--r-- 1 antmedia antmedia  172153 Mar  3 18:23 hs_err_pid30886.log
-rw-r--r-- 1 antmedia antmedia  164505 Mar  3 18:24 hs_err_pid31280.log
-rw-r--r-- 1 antmedia antmedia  148861 Mar  3 18:25 hs_err_pid31632.log
-rw-r--r-- 1 antmedia antmedia  158891 Mar  3 18:26 hs_err_pid31877.log
-rw-r--r-- 1 antmedia antmedia  156248 Mar  3 18:28 hs_err_pid32078.log
-rw-r--r-- 1 antmedia antmedia  157205 Mar  3 18:29 hs_err_pid32315.log
-rw-r--r-- 1 antmedia antmedia  250172 Mar  3 20:07 hs_err_pid32538.log
-rw-r--r-- 1 antmedia antmedia  206044 Mar  3 20:07 hs_err_pid46973.log
-rw-r--r-- 1 antmedia antmedia  226068 Mar  3 20:09 hs_err_pid47665.log
-rw-r--r-- 1 antmedia antmedia  207825 Mar  3 20:10 hs_err_pid48972.log
-rw-r--r-- 1 antmedia antmedia  212255 Mar  3 20:33 hs_err_pid5783.log
-rw-r--r-- 1 antmedia antmedia  242304 Mar  3 20:25 hs_err_pid777.log
-rw-r--r-- 1 antmedia antmedia  181275 Mar  3 20:33 hs_err_pid8738.log
-rw-r--r-- 1 antmedia antmedia  157187 Mar  3 20:34 hs_err_pid9180.log
-rw-r--r-- 1 antmedia antmedia  168797 Mar  3 20:35 hs_err_pid9485.log
-rw-r--r-- 1 antmedia antmedia  190964 Mar  3 16:50 hs_err_pid9609.log
-rw-r--r-- 1 antmedia antmedia  214222 Mar  3 20:39 hs_err_pid9765.log

The "Context has not been initialized" error seems to happen around the same time as the daemon stops responding. It's impossible to stop it via "service antmedia stop" - we must kill the PID.
The issue is very likely correlatated with someone trying to broadcast from Safari on an IPad (WebRTC Ingress) with the following user agent:
5.0 (iPad; CPU OS 12_4_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1

The server in question was crashing like crazy, and we moved that user to another server. The first server stopped crashing, the second server started.

 

Link to comment
Share on other sites

  • 0

Hi Konrad,

Thank you for reporting to us!

We need to check the beginning of Ant Media Server logs. Could you please send us a full of logs? You can find ant-media-server.log file under the /usr/local/antmedia/log folder. Also, please send hs_err_pid10035.log file. Are you using Adaptive bitrate?

Looking forward to hearing from you soon.

Best Regards,
Selim
Link to comment
Share on other sites

  • 0

We also started noticing the same error for last few days. Do we know what can cause it. When ever we see this error, ant media server getting restarted. see below for the logs.

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3].

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener initializeSSL

INFO: OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]

Mar 03, 2021 7:53:49 PM org.apache.coyote.http11.AbstractHttp11Protocol configureUpgradeProtocol

INFO: The ["http-apr-0.0.0.0-5080"] connector has been configured to support HTTP upgrade to [h2c]

Mar 03, 2021 7:53:49 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"]

Mar 03, 2021 7:53:49 PM org.apache.coyote.http11.AbstractHttp11Protocol configureUpgradeProtocol

INFO: The ["https-openssl-apr-0.0.0.0-5443"] connector has been configured to support negotiation to [h2] via ALPN

Mar 03, 2021 7:53:49 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.StandardService startInternal

INFO: Starting service [Tomcat]

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/8.5.58

Mar 03, 2021 7:53:49 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment

INFO: No global web.xml found

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 7:53:50 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 7:53:50 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 7:53:51 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"]

Link to comment
Share on other sites

  • 0
Selim, i've sent you the log file and a pid. We are still waiting for the user to reply with the exact device model they are using but it seems to be an old as hell iPad - the resolution we locked was 768x1024 so it's probably circa 2013-2014 at best. One of our staff noted we actually had an influx of crashes before as well when someone tried to stream from an old iPhone. It seems that these users are able to fairly consistently cause Antmedia daemon to stop responding just by streaming to it.

We have for now disabled this users account as  it was killing our production environment, and disabled streaming from iOS prior to v13 to prevent old devices from using the system.

Staff member tried streaming through 2.3.0 RC1 with his  IPhone XR (iOS 14) and the log is also flooding with these errors - albeit it didn't kill the daemon, so it seems to be recreatable by using Safari on an iPhone/iPad as WebRTC Ingress.



Link to comment
Share on other sites

  • 0

This issue is not resolved in the March 2nd build. We are also having substantial issues with audio not working for many broadcasters or cutting out.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
java.lang.NullPointerException
        at io.antmedia.enterprise.adaptive.AVFramePool.getAVFrame(AVFramePool.java:25)
        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.onOpusPacket(WebRTCEncoderAdaptor.java:1007)
        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.lambda$getAudioDecoderFactoryFactory$0(WebRTCEncoderAdaptor.java:960)
        at org.webrtc.BuiltinAudioDecoderFactoryFactory.onAudioPacket(BuiltinAudioDecoderFactoryFactory.java:41)
Exception in thread "network_thread - 27933"
# Fatal error in: gen/sdk/android/generated_builtin_audio_codes_jni/../../../../../../sdk/android/src/jni/jni_generator_helper.h, line 85
# last system error: 0
# Check failed: !env->ExceptionCheck()
# WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3].
Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]
Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener initializeSSL
INFO: OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]
Mar 08, 2021 12:48:16 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"]
Mar 08, 2021 12:48:16 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]
Mar 08, 2021 12:48:16 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Mar 08, 2021 12:48:16 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/8.5.58
Mar 08, 2021 12:48:16 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Mar 08, 2021 12:48:16 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Mar 08, 2021 12:48:17 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize
SEVERE: Spring context lookup failed, skipping spring component provider initialization.
Mar 08, 2021 12:48:18 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Mar 08, 2021 12:48:18 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize
SEVERE: Spring context lookup failed, skipping spring component provider initialization.
Mar 08, 2021 12:48:18 PM org.apache.catalina.core.ApplicationContext log
INFO: No Spring WebApplicationInitializer types detected on classpath
Mar 08, 2021 12:48:18 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize
SEVERE: Spring context lookup failed, skipping spring component provider initialization.
Mar 08, 2021 12:48:18 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"]
Mar 08, 2021 12:48:18 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.
(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

Link to comment
Share on other sites

  • 0

Thanks - some more logs from times it crashed/stopped responding, or marked SEVERE:

java.lang.NullPointerException
        at io.antmedia.enterprise.adaptive.AVFramePool.getAVFrame(AVFramePool.java:25)
        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.onOpusPacket(WebRTCEncoderAdaptor.java:1007)
        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.lambda$getAudioDecoderFactoryFactory$0(WebRTCEncoderAdaptor.java:960)
        at org.webrtc.BuiltinAudioDecoderFactoryFactory.onAudioPacket(BuiltinAudioDecoderFactoryFactory.java:41)
Exception in thread "network_thread - 27933"

#
# Fatal error in: gen/sdk/android/generated_builtin_audio_codecs_jni/../../../../../../sdk/android/src/jni/jni_generator_helper.h, line 85
# last system error: 0
# Check failed: !env->ExceptionCheck()
# WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

-----------------

Tons of this being spammed in the logs (thousands of entries), after we tried switching to Mono yesterday to try see if that resolves it.
[opus @ 0x7f7a7012a1c0] Error decoding a SILK frame.
[opus @ 0x7f7a7012a1c0] Error decoding an Opus frame.
[opus @ 0x7f7a7012a1c0] Error decoding a SILK frame.
[opus @ 0x7f7a7012a1c0] Error decoding an Opus frame.

Link to comment
Share on other sites

  • 0

Another log from another unresponsiveness/crash today (2.3.0 March 2nd build)

Mar 09, 2021 6:07:18 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [140,548,515,160,224] with wrapper [org.apache.tomcat.util.net.AprEndpoint$A$
        at javax.websocket.CloseReason.<init>(CloseReason.java:70)
        at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153)
        at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Mar 09, 2021 6:07:18 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process
SEVERE: Error reading request, ignored
java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [140,548,513,878,176] with wrapper [org.apache.tomcat.util.net.AprEndpoint$A$
        at javax.websocket.CloseReason.<init>(CloseReason.java:70)
        at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153)
        at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Link to comment
Share on other sites

  • 0

And just to be clear as to what we consider unresponsive - the criteria for our script that catches and logs these issues is HTTP accessibility to Antmedia, so we are testing http://localhost:5080 and it takes longer then 5 seconds to respond we kill the pid and restart it. During timeouts like this, we are unable to do 'service antmedia stop' as it just hangs there.

 

Link to comment
Share on other sites

  • 0

Unfortunately we haven't been able to identify anything in particular. The site just has performers streaming using WebRTC ingress and viewers using WebRTC Egress. It's random. Sometimes will happen 5x a day sometimes won't happen for 2 weeks. Sometimes there's 10 performers on a server and it's fine, sometimes with 1 it starts locking up.

The only crash we were able to identify was caused by Safari WebRTC Ingress on a  first gen Ipad Air but even when we blocked iOS < v13 we still have the daemon becoming unresponsive. The rest are using various devices, mostly chrome on pc, sometimes macs, sometimes android tablets or phones. Everything is done via browser.
This has been going effectively since we started using the software a year ago and the only way we can keep everything going is by running our timeout script on a cron every 2 minutes checking the status, killing the pid and starting the daemon again.  We've had alot of problems overall. Egress tends to be choppy for the first 15-20 seconds and then clears up - sometimes remains choppy but in some cases it is performer having terrible internet, 10 year old laptop or they are using wifi and apparently live in a Faraday cage. 

We suspected it might have to do with audio so we changed to Mono, which seemed to fix an issue with audio vanishing after a while but not the unresponsiveness. There's sometimes not even anything in the logs - it just becomes unresponsive and that's that. Most servers are 32 thread and 128-192gb of ram. We aren't hitting any resource limits. The amounts of viewers might be 50-100 on a server during peak. 

Link to comment
Share on other sites

  • 0

Has any progress on this been made?

Every time we crash we get this in the logs. Our error logs are several gigabytes in size...

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I found this, maybe will help.
Link to comment
Share on other sites

  • 0

Hi Konrad,

Have a good day.

Every time we crash we get this in the logs. Our error logs are several gigabytes in size...

I'm sorry for the late response. Actually, I discussed this issue with my colleagues, but we couldn't reproduce this issue. Is there any specific reproduce scenario for this issue? 

Thank you for reporting this warning log issue. I added an issue for you -> https://github.com/ant-media/Ant-Media-Server/issues/3184

Best Regards,
Selim

Link to comment
Share on other sites

  • 0

We are still experimenting with this. Thus far, we have seen one person be fairly consistently able to crash teh daemon by broadcasting from a very old macbook with Chrome 67. Not every time, but often. We have also seen this happen from some phones but also not always. Seems to be related to older apple devices we think. Right now we are playing wtih resolutions to see if this is maybe causing issue - basically going off 'what are we doing differently then everyone else'

 

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...