RuntimeError: SPF processing a request before App server is started

Questions and Help
#1

I kept on getting this error after using sanic-cors:

RuntimeError('SPF processing a request before App server is started.')

Is there anyway to deal with this? At least I want to be able to catch this and not show a dreaded 503 page to the enduser — when it appears to be “expected behavior” when using this library.

I know that it only shows up very briefly when the application starts, but I’d still rather not show it even if only 5 people see.

#2

Hi @smlbiobot

I think there might be a misunderstanding here. This error message is emitted by Sanic-Plugins-Framework, not Sanic-CORS. And seeing this message is certainly not “expected behavior”. It is a sign that something went wrong, that is why an error is thrown. Are you able to link to any discussion topics where it is implied that it is expected behavior?

Sanic-Plugins-Framework has a mechanism to detect if incoming requests are entering the application before the Sanic server has started. At that time, the sanic application, its routes, and its middleware are in an “undefined” state. Only after the Sanic server emits a “server_started” event, should requests be processed. If a request was allowed through before that, invalid results could be produced.
Sanic-Plugins-Framework listens for the “server_started” event, and uses that to disable this error message after the server starts.

This mechanism can be fragile, here are some times it has failed:

  1. When using Sanic in ASGI Mode, the server emits an “after_server_start” event, but never a “before_server_start” event, which confuses SPF. This was fixed in SPF in 0.8.2 in Aug 2019.
  2. When using the test-client within sanic to execute async requests against a non-running sanic server. This was fixed in 0.9.3 in July 2020
  3. When manually executing an async server instance created using app.create_server(), and similar circumstances more recently here. This never emits any “server_start” event, so breaks SPF (and other plugins) on startup. There are workarounds here and here.

In each of these situations, the error message is/was not expected behavior, and steps were taken to prevent it from happening.

Are you able to give us more details about your application and method of executing Sanic? With more information we might be able to get to the bottom of this together.

I think there is a small chance, under very high load, a race condition may cause this error to be thrown for few requests at startup. There would be a period of between 1 and 10 milliseconds after the server starts up, but before SPF starts to allow requests through. If you have very high load, in the order of hundreds of requests per second on your application before it starts up, then one or two might make it through before requests are allowed.
Does that correspond with what you are seeing?

1 Like
#3

Update:
@smlbiobot
I’ve just released a new version of SPF, v0.9.4.
It has a fix for the potential race condition outlined in my previous post.
And I’ve changed the Error from RuntimeError to Sanic ServerError which may be easier to monitor and catch, I’m not sure, I haven’t tested that.

#4

Thanks for the reply! I’ve looked into your solutions but I am not starting the app with app.create_server()

I am get this errors under these conditions (these are the only two modes that I use in local and also the servers:

Directly as a module on dev:

python -m sanic cr.app.app --host=0.0.0.0 --port=8000 --workers=2 --debug

Run in gunicorn:

gunicorn \
    cr.app:app \
    --bind 0.0.0.0:8000 \
    --workers=4 \
    --backlog 2048 \
    --worker-class sanic.worker.GunicornWorker \
    --timeout 30 \
    --keep-alive 30 \
    --max-requests 30000 \
    --max-requests-jitter 10000 \
    --worker-connections 1000

I’ll take a look and see if the new version of SPF fixes this issue, and/or try to see if I can at least catch the error.

#5

FWIW, these are the versions I’m on:

sanic==20.6.3
Sanic-Cors==0.10.0.post3
Sanic-Plugins-Framework==0.9.3
#6

@smlbiobot
Did the change in SPF v0.9.4 help to fix the problem you were seeing?

#8

@ashleysommer I think that it is working — for sure I haven’t seen the same errors as before now. So thank you!