Sanic worker with python-socketio crashes when the client does a hot-reload

Questions and Help
#1

Hi,

I’m using python-socketio with Sanic, and having a client (front-end) running ReactJS (using react-boilerplate.

The Sanic app sometimes crashes when the client performs a hot-reload. Here is the error logs retrieved from journalctl, the service’s name is Ora socketio:

Oct 31 16:04:39 ora-backend systemd[1]: Stopping Running Ora socketio...
Oct 31 16:04:39 ora-backend python3.7[15005]: [2019-10-31 16:04:39 +0000] [15005] [INFO] Stopping worker [15005]
Oct 31 16:04:39 ora-backend python3.7[15005]: [2019-10-31 16:04:39 +0000] [15005] [ERROR] Experienced exception while trying to serve
Oct 31 16:04:39 ora-backend python3.7[15005]: Traceback (most recent call last):
Oct 31 16:04:39 ora-backend python3.7[15005]:   File "/root/venv/lib/python3.7/site-packages/sanic/app.py", line 1135, in run
Oct 31 16:04:39 ora-backend python3.7[15005]:     serve(**server_settings)
Oct 31 16:04:39 ora-backend python3.7[15005]:   File "/root/venv/lib/python3.7/site-packages/sanic/server.py", line 825, in serve
Oct 31 16:04:39 ora-backend python3.7[15005]:     loop.run_until_complete(asyncio.sleep(0.1))
Oct 31 16:04:39 ora-backend python3.7[15005]:   File "uvloop/loop.pyx", line 1415, in uvloop.loop.Loop.run_until_complete
Oct 31 16:04:39 ora-backend python3.7[15005]: RuntimeError: Event loop stopped before Future completed.
Oct 31 16:04:39 ora-backend python3.7[15005]: Traceback (most recent call last):
Oct 31 16:04:39 ora-backend python3.7[15005]:   File "app_socketio.py", line 6, in <module>
Oct 31 16:04:39 ora-backend python3.7[15005]:     app.run(**SOCKETIO_RUN_CONFIG)
Oct 31 16:04:39 ora-backend python3.7[15005]:   File "/root/venv/lib/python3.7/site-packages/sanic/app.py", line 1135, in run
Oct 31 16:04:39 ora-backend python3.7[15005]:     serve(**server_settings)
Oct 31 16:04:39 ora-backend python3.7[15005]:   File "/root/venv/lib/python3.7/site-packages/sanic/server.py", line 825, in serve
Oct 31 16:04:39 ora-backend python3.7[15005]:     loop.run_until_complete(asyncio.sleep(0.1))
Oct 31 16:04:39 ora-backend python3.7[15005]:   File "uvloop/loop.pyx", line 1415, in uvloop.loop.Loop.run_until_complete
Oct 31 16:04:39 ora-backend python3.7[15005]: RuntimeError: Event loop stopped before Future completed.
Oct 31 16:04:39 ora-backend systemd[1]: ora_socketio.service: Main process exited, code=exited, status=1/FAILURE
Oct 31 16:04:39 ora-backend systemd[1]: ora_socketio.service: Failed with result 'exit-code'.

Do you know what the causes of this are ?

Here is the code I use for the Sanic app and the client:

Sanic app
Version: 19.6.3

# app_socketio.py
SOCKETIO_RUN_CONFIG = {
    "host": "0.0.0.0",
    "port": 8080,
    "debug": False,
    "access_log": False,
    "workers": 1,
    "protocol": WebSocketProtocol,
}
app.run(**SOCKETIO_RUN_CONFIG)

Client (ReactJS)

const socket = socketIOClient('http://157.230.253.130:8080', {
      transportOptions: {
        polling: {
          extraHeaders: {
            Authorization: `Bearer ${access_token}`,
          },
        },
      },
      transports: ['polling'],
    });
#2

It states the problem in the traceback. There is probably something going on with socketio, but there’s no good way to tell.

#3

I see. I’m going to debug a little more and see if I could get more detailed logs.

Followed up question, is there a way to recover the Sanic worker for python-socketio after crashing ?

Right now, when the python-socketio worker crashes for any reasons, it returns 503 response and unable to process any requests, until I have to manually kill the process and restart it.

#4

After debugging and running the worker in debug mode, it seems when the worker crashes, it only rejects new connections but the current clients still can exchange events with the worker.

However, this issue has been resolved after I put the Sanic app behind Nginx.