Haproxy maxconn

[Bob Perry]

Hi,

Hi fairly new to Ant Media  clustering. We're running on vmware Cisco UCS/Pure Storage. Each node has 12vCpu and 24GB ram. LB and Mongo has 4 CPU and 16GB Ram. 

I have a cluster with 5 origin nodes and 3 edge nodes. haproxy maxconn is at default 2000. We're running into an issue when we reach 110 streams (RTMP PUSH) where we start to get the stream will start playing etc.. message and streams do not play. In order to stabilize system we backed off the RTMP PUSH to 86 streams. 

Will increasing the maxconn setting solve this issue? Can you recommend a value for this setting for a cluster this size? Looking to support upwards of 200 RTMP streams. Thanks!

 

Edited July 5, 2022 by Bob Perry

[Mohit]

22 hours ago, Bob Perry said:

Hi,

Hi fairly new to Ant Media  clustering. We're running on vmware Cisco UCS/Pure Storage. Each node has 12vCpu and 24GB ram. LB and Mongo has 4 CPU and 16GB Ram. 

I have a cluster with 5 origin nodes and 3 edge nodes. haproxy maxconn is at default 2000. We're running into an issue when we reach 110 streams (RTMP PUSH) where we start to get the stream will start playing etc.. message and streams do not play. In order to stabilize system we backed off the RTMP PUSH to 86 streams. 

Will increasing the maxconn setting solve this issue? Can you recommend a value for this setting for a cluster this size? Looking to support upwards of 200 RTMP streams. Thanks!

 

Hi Bob,

How are you?

Thank you for writing to us and I welcome you to our community.

I think 5 origin nodes each of 12 cores should be able to handle 200 broadcast. 

Can you please tell me how you are broadcasting and playing the streams?

Also are you using multiple Adaptive settings on Origin nodes?

 

--

Best Regards,

Mohit Dubey

[Bob Perry]

Hi Mohit,

Thanks for responding.  
We are using kiloview encoders RTMP push for WebRTC. Using HA-Proxy version 2.0.28-1ppa1~focal, Ant Media Enterprise Edition 2.4.2.1 20220201_1956, and MongoDB server version: 5.0.6
We are not using adaptive settings ie: nothing configured on settings page.  
We only have H.264 checked under WebRTC codec support. 
Originally our team was configuring streams in the gui and selecting RTMP Push on the encoders but we abandoned that method, deleted all the streams from the gui and all push configuration is now done on the kiloview. Also our team was trying to send rtsp from a few IP cameras but we have since changed those to RTMP push as well. 

Is it safe to delete the local webrtcappee.db and liveapp.db from the nodes? Reason I ask is when the team was trying to setup the RTSP cameras they received a message to select an individual server. This could be nothing but figured I'd ask. 

I see response errors on the haproxy stats page for backend_rtmp, not sure if that is a maxconn setting issue? Here's my haproxy.cfg with server names etc. removed

global
    log 127.0.0.1 local0 notice
    maxconn 5000
    user haproxy
    group haproxy
    tune.ssl.default-dh-param 2048
defaults
    log global
    mode http
#    option forwardfor
    option http-server-close
#    option httplog
    option dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  100000
    timeout tunnel  2h  #this is for websocket connections, 2 hours inactivity timeout
    timeout client-fin 5000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

listen stats # Define a listen section called "stats"
  bind :6080
  mode http
  option httplog
  stats enable  # Enable stats page
  stats hide-version  # Hide HAProxy version
  stats realm Haproxy\ Statistics  # Title text for popup window
  stats uri /haproxy_stats  # Stats URI
  stats auth ******:********  # Authentication credentials

frontend rtmp_lb
    bind *:1935
    mode tcp
    option tcplog
    default_backend backend_rtmp

backend backend_rtmp
    mode tcp
    server ams1 **************:1935 check  # Ant Media Server instance 1
    server ams2 **************:1935 check  # Ant Media Server instance 2
    server ams3 **************:1935 check  # Ant Media Server instance 3
    server ams4 **************:1935 check  # Ant Media Server instance 4
    server ams5 **************:1935 check  # Ant Media Server instance 5
   # you can add more instances

frontend http_lb
  bind *:80
  bind *:5080
  mode http
  option httplog
# lines added for forwarding
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
#  http-request add-header X-Forwarded-Proto:\ http
  default_backend backend_http

frontend frontend_https
  option httplog
  bind *:443 ssl crt  /etc/ssl/certs/**************.pem
  bind *:5443 ssl crt /etc/ssl/certs/**************.pem
#  http-request add-header X-Forwarded-Proto:\ https
  http-request redirect scheme https unless { ssl_fc }
  default_backend backend_http

backend backend_http
#  leastconn
  # below line forwards http requests to https, if you do not have SSL termination, remove it
  redirect scheme https if ! { ssl_fc }
  # below line provides session stickiness
  cookie JSESSIONID prefix nocache
  server ame1 **************:5080 check cookie ame1  #if you do not use session stickiness, remove cookie ame1
  server ame2 **************:5080 check cookie ame2  #if you do not use session stickiness, remove cookie ame2
  server ame3 **************:5080 check cookie ame3  #if you do not use session stickiness, remove cookie ame3
#  server ame4 **************:5080 check cookie ame4  #if you do not use session stickiness, remove cookie ame4

 

[Bob Perry]

restarted Haproxy yesterday and these response errors continue to increase.