Quality degradation when using AntMedia (same wifi, same device, same time)

[Joe Suh]

We use Antmedia to add a graphic overlay to our user's stream. They can choose to stream their video on Facebook directly, or through our Antmedia instance to add the overlay.

We have had several cases where the video and audio quality degrade quite a bit the minute the user switches to streaming through our Antmedia.

Here is one example using Antmedia: https://www.facebook.com/ZBoutiqueMadison/videos/333563245285874

It was very glitchy and pixelated as you can see from the video. User reported a lot of lag as well. They then went live immediately afterwards with the same phone and same internet connection directly on Facebook, and the quality was much better:

https://www.facebook.com/ZBoutiqueMadison/videos/991270501601476

How can we go about troubleshooting the Antmedia live stream? I realize there is some overhead with the graphic overlay, but the quality difference is night and day. And we really need to get close to the quality level our users get when they go live directly on Facebook.

Thanks

Joe

[Mohit]

Hi,

Can you let us know about the Server specifications and the load that overlay is creating.

We can try and understand this better.

--

Best Regards,

Mohit Dubey

 

[Joe Suh]

Hi Mohit

Can you let us know what info you would like about the server? Or we can even give you temp IAM access to our AWS if that would be helpful to your team?

 

The overlay is created on the client’s side, so there is no additional load at server’s side. 

 

Joe

[Joe Suh]

Hi Mohit, 

We are using a c5.xlarge instance for this.

Joe

 

[Selim Emre]

Hi Joseph,

Have a good day.

Many things can cause this issue. It's hard to understand without looking at your logs. Is it possible to add your logs in here? You can find ant-media-server.log file on /usr/local/antmedia/log folder.  

Also please check this blog post for more detail: https://antmedia.io/webrtc-video-play-pixelating-blurring-issues-troubleshooting/

Looking forward to hearing from you soon.

Best Regards,

Selim

[akbar butt]

Hi Selim,

You can review the latest log files from the following URL: 

https://drive.google.com/drive/folders/1VWcEgsObkPr6EZ6sqrntcmnHJ3QNxNR9?usp=sharing

Thanks

[Selim Emre]

Hi,

Have a good day. 

I have checked your logs. As I saw you're using old version of Ant Media Server v2.3.3.1. I'm recommending using latest version. We improved the RTMP Endpoint feature on new versions.

I saw that there are some packet losses in your streamings. Are you sure your Ant Media Server Instance is close to your publisher network? 

Looking forward to hearing from you soon.

Best Regards,

Selim

[akbar butt]

Hi Selim,

We will upgrade it in 2 to 3 days and we will update you after that.

We have tried to update in recent past and our streaming stopped working. Do we have to re- register the streams after upgradation.

Thanks

Akbar

 

[Selim Emre]

Hi Akbar,

No need to re-register your streams. If you use "-r true" parameter in installation, your app details will be stored.

Please check it for more details: https://github.com/ant-media/Ant-Media-Server/wiki/Installation#update-over-older-version

Best Regards,

Selim

[akbar butt]

Hi Selim,

We have upgraded to Enterprise edition 2.4.x but after that our video did not render in iframe (play.html), We have reverted back so that it will keep working for the LIVE customers. We did not change anything in the code. Will you please update us about it?

Thanks

Akbar

 

[akbar butt]

Hi Selim,

We have upgraded to Enterprise edition 2.4.x but after that our video did not render in iframe (play.html), We have reverted back so that it will keep working for the LIVE customers. We did not change anything in the code. Will you please update us about it?

Thanks

Akbar

 

[Selim Emre]

Hi Buttar,

Have a good day. Please upgrade to the latest stable version. v2.4.2.1 is available on your download page.

Btw, If you send error details to the support channel(support@antmedia.io), they can give you more detail about your error. 

Looking forward to hearing from you soon.

Best Regards,

Selim

[akbar butt]

Hi Selim,

ok, I will confirm the downloaded version.

We are getting the error "No stream exists", I have verified from the dashboard and stream id exists.

Thanks

Akbar

 

[Selim Emre]

Hi Akbar,

"No stream exists" error is so generic. I need to check your logs. Could you please attach new release error logs as well? 

Best Regards,

Selim

[akbar butt]

Hi Selim,

We are deploying the latest version on a separate instance so that it will not affect the LIVE, I will update you about it during next week.

Thanks

Akbar

 

[akbar butt]

Hi Selim,

We have upgraded to "2.4.2.1 20220201_1956" and video streaming is working now on the latest version. Let we monitor and if we found an issue with the quality, We will share that with you.

Thanks

Akbar

 

[Joe Suh]

Thanks for your help Selim.

So far so good with the customers using it. We'll know more within the next few days if the customers that previously had issues still face them with this new version. Happy to report back in this thread...

Joe

 

[akbar butt]

Hi Selim,

Is there any way to automatically detect the streaming quality?

Thanks

Akbar

 

[Selim Emre]

Hi Akbar,

Have a good day.

Is there any way to automatically detect the streaming quality?

I couldn't get it clearly. Could you please give me more detail about your question? If you meant Facebook, it automatically detects the streaming quality. No need to add adaptive bitrate in Ant Media Server. 

Best Regards,

Selim

[Joe Suh]

Hi Selim,

What Akbar and I are looking for is a way to quantitatively or automatically evaluate the quality of the video stream. Right now we are manually watching videos and trying to visually compare before-and-after to see if there is a difference in quality.

Any tips to help us with this are appreciated

Thanks

Joe

[Selim Emre]

Hi All,

As I understand you are using WebRTC for publishing. We are providing some details about the video stream. You can check it. I'm sure it will help your issue. You can check WebRTC JS SDK for more detail: https://github.com/ant-media/StreamApp/blob/master/src/main/webapp/js/webrtc_adaptor.js#L1752

I have also attached example stream details.

I hope, I could help you.

Best Regards,

Selim

 

 

[akbar butt]

Hi Selim,

We are already saving these stats after every 5 seconds for each video. Sometimes we get the "qualityLimitationReason" as bandwidth then we can consider it as the problem occurs due to the user's internet provider. But sometimes we will not have anything in the qualityLimitationReason and the user reports that the video quality is not good. In this case we will watch the video to identify the problem.

Mostly when video quality is not good using our system, The user tries to go LIVE directly on FB and that was much better as compared to our system. So we want to identify the reason why our stream is not as good as FB. We need to know it in order to adjust things.

We have also verified the network measurement tool provided by ant media with one of the users. The measurement tool indicates that everything is fine but the quality of the video stream was not good at that time using the same browser and internet.

If you have any idea about it then please update us accordingly.

We have upgraded the server and generally we are better till now, We will share the details if we notice any quality issue after this upgrade.

Thanks

Akbar

[Ant Man]

Hi Akbar

Just throwing my two cents worth  into this thread. 

If you are streaming outbound via WebRTC to end clients - you need to understand that each person receiving the feed will probably have a different ISP and route back to your server. 

I’m going to assume you’re streaming from and to one geographical location — correct?

Did you enable MBR/ Adaptive Bitrate on the AMS?

Is your publishing to AMS from the same Geo location and same/similar network that hosts your AMS ingest?

Do you have enough resources on that server under the load you anticipate (anticipated viewership)? The AMS dashboard is very good to show you your resources at idle and then under load. If you are at 40% with only 5 connections - then you’ll have the server on its knees with 50 connections — 

Do you have enough ISP backbone for each individual connection to that AMS instance?

When you stream to a CDN like FBL or YouTube etc - you’re publishing to their massive cloud network with thousands of servers that can do load balancing etc etc - hence the better quality. 

Look at what you have internally and then go from there. 

I’ll let the folks from Ant take it from here - but my experience is if you do not have enough resources internally and externally - you will have a depleted or failed broadcast from a self hosted standpoint. 

Regards

Sean

[akbar butt]

Hi Sean,

Please review the following answers:

If you are streaming outbound via WebRTC to end clients - you need to understand that each person receiving the feed will probably have a different ISP and route back to your se

1. We are streaming on FB using RTMP links.
2. We have used webrtc for original streaming then we will get the ant media rtmp urls and registered that with FB page or group    

I’m going to assume you’re streaming from and to one geographical location — correct?

1. Yes

Did you enable MBR/ Adaptive Bitrate on the AMS?

1. We did not add any new bit rate under the "Adaptive Streaming section"
2. We have tried that in recent past but it will increase the COU utilization. It's CPU intensive so we have removed the adaptive bit rates.

Is your publishing to AMS from the same Geo location and same/similar network that hosts your AMS ingest?

1. Our code is on AWS  "Oregon (us-west-2)" region while Ant media server is hosted on "N.Virginia (us-east-1)" region
2. Our code initiate the stream using HTML5 canvas.

Do you have enough resources on that server under the load you anticipate (anticipated viewership)? The AMS dashboard is very good to show you your resources at idle and then under load. If you are at 40% with only 5 connections - then you’ll have the server on its knees with 50 connections — 

1. Till now everything is fine on Ant media AWS instance.
2. the last one week Ant Media AWS instance COU utilization image is attached

Do you have enough ISP backbone for each individual connection to that AMS instance?

1. I did not understand it properly
2. Obviously We did not control the individual ISP from different customers
3. Our general behavior is to publish stream on Facebook. The problem occurs when quality of video is not good using our Ant media implementation. But the quality is fine when user post directly on FB live using the same phone, internet etc.

Thanks

Akbar

 

 

 

[Selim Emre]

Hi guys,

@Sean thanks for the contribution.

For WebRTC streaming, network stability is so important. Maybe media constraints parameter modifies can help your stream quality. For example, if you publish on a mobile phone, then you can use low-resolution parameters(it doesn't mean low quality). 

Here are some guides for media constraints usage: 

https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints

https://webrtc.github.io/samples/src/content/peerconnection/constraints/

I hope, I could help you.

Best Regards,

Selim