Event details: On 27/05/2021 users experienced degraded service on app.surfly.com. 

There were some unexpected session disconnects in the EU region and inability to start sessions globally for short durations a few times between 09:59 CEST and 13:02 CEST on 27/05/2021. Slowness of app.surfly.com was also experienced during the same durations.

Root Cause Analysis: With the start of business hours in European region and the ongoing business hours in the Asia region, the usage of app.surfly.com increased rapidly. The load was higher than expected.

• At the time of receiving first alerts (09:59 CEST), the SSL termination endpoint was unable to carry the load, enabling the load balancer to fail-over to the backup server. 
• This kept happening and was causing some active session disconnects for users in the European region. 
• We started investigating the issue and took various measures but the issue kept coming back.
• It was discovered that the issue was occurring due to an internal limit to the maximum number of allowed HTTPS connections to the servers.

Resolution: 

• We've increased the connection limit of our HAProxy server to a number more suitable for the size of the server thereby massively increasing the load that it can carry.