Websocket crashing under heavy load

Questions and Help
#1

Hi guys, I am facing the below exception when bombarded with many join calls in my production Sanic application. The number of join calls are around 200-250 requests per sec.

message:
Fatal error on transport TCPTransport (error status in uv_stream_t.shutdown callback)
protocol: <sanic.websocket.WebSocketProtocol object at 0x7ff5a8584980>
transport: <TCPTransport closed=False reading=False 0x55c6640c42f0>
,stack_trace: OSError: [Errno 107] Transport endpoint is not connected

Can you guys help me in understanding the exception message? What does this mean and what could be the possible cause for throwing the exception?

Below are few more details, might be helpful to understand the issue better.
Sanic version: 19.3.1
UV Loop version: 0.12.2
Websocket version: 0.6

If you need any more information, I am happy to help you with that.
Thanks!

#2

Those are some old packages you are using. Websockets went through a couple of major changes since then, and we no longer support Sanic v19.3.

200-250 connections per second for how long? And how long are they staying open? How many processes. Perhaps you are exhausting numbers of open connections.

I would say first thing to do is upgrade your packages.

#3

Sure! Can you please suggest me the stable combination which I should go for?
We were running 4 processes and the traffic came in for around 1-2 mins. Then after that this error showed up. Also I want to understand this exception, what does this mean? Can you also help me with that?

#4

Supported versions of Sanic are 19.12LTS and 20.6.

Your machine is reporting a ENOTCONN message, which means that the socket is not connected. As to why, that would be hard to know from this limited information. Perhaps you exhausted too many connections and cannot receive more? Perhaps a connection got closed prematurely? Depending on your proxies (or the client’s) maybe an intermediary closed it for inactivity?