High CPU usage on server

[Divyanshu Kalra]

We have a conference app setup, and if 2 people connect, and i share my screen, and then unshare and share again a couple of times, CPU usage of server jumps to 50%+. (CPU config: 4cores, 16gb RAM, AWS). I am using ant-media-server-enterprise-2.1.1-20200902_1011. I am attaching the output of /rest/thread-dump-json if that helps. Also once we end the call, the CPU usage goes back to single digits.

 

dumpThread.json

[Nagendra Kumar1648581702]

Hi Divyanshu,

   We are using screenshare with antmedia server. I had the issue of client cpu usage but when we reduced the fps then everything was very good and no issue so far. Server side I am using 4 core 15GB ram all good so far.

Thanks

Nagendra

[Divyanshu Kalra]

Hey, I am not talking about client cpu usage, we are facing high level of cpu usage on the server. It is normal in the start, but if i toggle screenshare 3-4 times, it suddenly shoots up to 50 - 60%.

 

 

[Selim Emre]

Hi kalradivyanshu,

Did you enable Adaptive option? Also which Instance type are you using? Could you please share your console/error logs(based in /usr/local/antmedia/log folder)? 

Best Regards,

Selim

 

[Divyanshu Kalra]

Hey,

I have adaptive option disabled. t2.xlarge is the aws instance. I have attached all the logs.

 

 

errorLogs.txt

threadRaw.txt

threadinfo.txt

accessLogs.txt

threadJSON.txt

[mekya]

Hi,

Just a reminder,  media streaming and especially the WebRTC streaming is a process intensive operation. It's always good to use Computed Optimized Instance types.

For AWS, you can use C family. 

Secondly, t2 family provides variable CPU and each instance has some CPU credits. Its CPU credit is 54 per hour. 

A CPU Credit provides the performance of a full CPU core for one minute. So baseline performance of t2.xlarge is 54/60 of a CPU core. 

I mean its baseline is almost 1 core CPU. If you don't use the credits, it's accumulated and you can have some spike up to 4 core CPU. 

If you consume all the credits, you can get less CPU than the baseline.  For more information, please take a look the https://aws.amazon.com/ec2/instance-types/#Burstable_Performance_Instances

Please try with C5 or C6 type instance having 4 cores CPU and let us know the results.

Regards,

A. Oguz  

[Divyanshu Kalra]

Hey, I shifted to digitalocean, we are now using compute optimized droplet (CPU-Optimized droplet), and still were able to recreate the issue. Can you please look into this?

[mekya]

Hi Divyanshu,

I just need to re-produce the issue on my end. 

Could you please let us have the exact re-produce scenario step by step?

Regards,

A. Oguz 

[Divyanshu Kalra]

 I created a small POC:

https://drive.google.com/file/d/1pCE12ZM1_1Ok5PkItfUmOcbqsrEafz8A/view?usp=sharing

I was able to recreate the issue when i used 2 laptops, and connected (using ngrok). Turn screensharing on and off a couple of times, and you will see cpu usage spikes to as high as 60%.

[mekya]

Thank you. 

I'll try to reproduce on my end. Let me confirm the reproduce scenario. There are some missing parts for me. 

1. Run Ant Media Server in a 4 Core CPU optimized instance in Digital Ocean

2. Deploy the files at antBug.zip to Ant Media Server

3. Turn screen sharing on and off a couple of times

Expected Behavior

CPU usage should be about (let's say) %15

Actual Behaviour

CPU usage goes to %60

I'm not sure about why and how to use ngrok in this scenario above.

Secondly, we encounter that TCP candidates sometimes cause high CPU usage.  Closing TCP cpu candidates resolves unexpected CPU usage. This issue will be investigated. As a workaround, you may try disabling TCP candidates by putting following properties in /usr/local/antmedia/webapps/{YOUR_APP}/WEB-INF/red5-web.properties file and restart the service

settings.webrtc.tcpCandidateEnabled=false

Regards,
A. Oguz

[Divyanshu Kalra]

Will try that, ngrok will be used if you deploy the demo on localhost, no need to use if you are using a server.

[mekya]

Before trying to it on my end. Just let me ask something. 

You're trying it on localhost are and you also capturing or playing the video on the same host? If yes, capturing video means rendering and encoding the video, playing the video means decoding and rendering the video. These are also process intensive operations for the processing unit. 

Regards,

A. Oguz

 

[Divyanshu Kalra]

I am just showing the localstream in the preview. I doubt that will take the servers cpu.

[Divyanshu Kalra]

I tried the settings.webrtc.tcpCandidateEnabled=false, and it still spiked to 30%+, and stayed there after 10 - 12 mins of call between 2 people with screen shared. Attaching the corresponding htop, when the cpu spiked.

 

 

[mekya]

Yes, showing a local stream takes the server's CPU. 

Anyway,  I have not had time to test your case yet. Let's make things clear . So the following re-produce scenario is true for you? Because it's not clear for me to test on Digital Ocean or local computer or AWS. Let's agree the re-produce scenario 

1. Run Ant Media Server in a 4 Core CPU optimized instance in Digital Ocean

2. Deploy the files at antBug.zip to Ant Media Server

3. Turn screen sharing on and off a couple of times

4. Wait for 10-12 minutes

Expected Behavior

CPU usage should be about (let's say) %15

Actual Behaviour

CPU usage goes to %60

[Divyanshu Kalra]

Yeah, but have more than 2 publishers, one sharing their screen, so total 3 streams (2 camera, 1 screen).

[mekya]

We'll investigate this case as an issue. Because if we don't schedule, we will burn out. I've almost experienced that a couple of times. 

 

https://github.com/ant-media/Ant-Media-Server/issues/2566

 

[Selim Emre]

Hi kalradivyanshu,

Have a good day. I have tried your use case. But I couldn't achieve.

I need to clear about below things:

1- Are you sure about settings.webrtc.tcpCandidateEnabled=false with right application?

2- Did you restart AMS, when you added this parameter in settings?

Looing forward to hearing from you soon.

Best Regards,

Selim

 

[mekya]

Hey Selim,

Have you tried the exact same scenario with html and js files that Divyanshu provided? Maybe there is a problem in js files that trigger this problem. 

Could you also let us know the results of your tests and steps you've followed?

Regards,

A. Oguz

[Selim Emre]

Hi Ahmet,

Yes, I tried with provided HTML and js files. 

Here are the steps:

1- I created 4 core Instance on Digital Ocean and Installed AMS 2.1.1 version(added settings.webrtc.tcpCandidateEnabled=false version).Also, I enabled SSL.

2- I installed and configured ngrok for publishing localhost HTML and js files.

3- I published the stream with ubuntu 18.04 and Windows 10 machines with 15 min.

4- I see everything seems normal. Also, I attached htop and other detail images.

 

 

 

 

error.log

console.log

[mekya]

I see. Thank you. 

 @Divyanshu Kalra 

We cannot create the reproduce scenario. Are we missing something?