Auto restarting Sanic server on touch of a specific file

[cx42net]

Following up on the discussion I had here (Restarting sanic from an external python script), I wanted to stop using the auto_reload option from Sanic and instead, control when the server would restart.

The main reason for that is simple: If an update made was by changing the database model, the server would fail until the DB migration was done.

Right now, my updating script works as follows:

When changing the code in my “prod” branch:

1. The code is automatically pulled on the production server.
2. Any package present in requiremetns.txt are automatically added or updated
3. The database is updated, using Alembic

Previously, this was causing trouble because:

• If any new package was not installed, Sanic would restart and fail because of the missing package.
• The restart would succeed, but the database model present in the code would not match the database since the upgrade command was not sent yet, causing SQL issues until step 3 was reached.

In order to avoid this, I’ve made a script that listens to one specific file modification time (“reload”), and only restarts Sanic when that file is touched.

With this, I’ve added a 4th step on my above list, which is simply to do a touch on that file as the last step, triggering the restart when everything is ready.

Here’s the code I’ve come up, please share your insight, if you have any better suggestions:

from sanic import Sanic
from sanic.log import logger
from watchdog.events import FileSystemEventHandler
from watchdog.observers import Observer
import os, signal


class ModificationEventLoader(FileSystemEventHandler):
    def __init__(self, handler):
        self.handler = handler

    def on_modified(self, event):
        if event.is_directory:
            return

        self.handler.restart()


class AutoReload:
    def __init__(self, auto_reload):
        self.first_loop = True
        self.is_restarting = False
        self.observer = None

        if auto_reload:
            logger.debug('Auto reload is already enabled, no need to setup the watchdog')
            return

        reload_file_path = os.path.join(Sanic.get_app().ctx.root_path, 'reload')
        if not os.path.isfile(reload_file_path):
            logger.info('Missing reload file. Auto reload will be disabled')
            return None

        self.observer = Observer()
        self.observer.schedule(ModificationEventLoader(self), reload_file_path, recursive=False)
        self.observer.start()

    def restart(self):
        self.is_restarting = True
        os.kill(os.getpid(), signal.SIGINT)

    def run(self):
        if self.first_loop:
            self.first_loop = False
            return True

        if self.is_restarting:
            self.is_restarting = False
            return True

        if self.observer:
            self.observer.stop()
            self.observer.join()

        return False

In order to make this to work, you’ll have to wrap your app.run(**options) to:

params = {
    'host': '127.0.0.1',
    'port': 8000,
    'debug': True,
    'access_log': True,
    'auto_reload': False
}

auto_reloader = AutoReload(params['auto_reload'])
while auto_reloader.run():
    # Running server
    app.run(**params)

The AutoLoader.run function will only return True at the first loop (starting) and when the “reload” file was changed, otherwise, it will return False (and stop the watchdog).

Note: This requires the watchdog package to handle the changes asynchronously. I’ve looked onto how Sanic handles file change when auto_reload is set to True, but it’s a while True with a time.sleep() which blocks the rest of the code. That wasn’t possible here.

Hope this helps!

[cx42net]

For those interested, my script executing the steps is another Sanic application called by the Github webhook on push to the production branch, here’s the code:

from sanic import Sanic
from sanic.response import text, empty
from sanic.exceptions import InvalidUsage, ServerError
from hmac import HMAC, compare_digest
from hashlib import sha256
import subprocess, os


GITHUB_WEBHOOK_KEY = 'your-webhook-key'
GITHUB_REPOSITORY = 'user/repo'
GITHUB_BRANCH = 'prod'
ROOT_PATH = '/var/www/server'

app = Sanic("DeployServer")


@app.post("/webhook/github/push")
async def github_push(request):
    received_signature = request.headers.get('X-Hub-Signature-256')
    if not received_signature or received_signature.find('sha256=') == -1:
        raise InvalidUsage('Invalid event.')

    received_signature = received_signature[received_signature.find('sha256=') + 7:]
    expected_signature = HMAC(key=GITHUB_WEBHOOK_KEY.encode(), msg=request.body, digestmod=sha256).hexdigest()
    if not compare_digest(received_signature, expected_signature):
        raise InvalidUsage('Invalid event.')

    if request.headers.get('X-Github-Event') == 'ping':
        return empty()
    elif request.headers.get('X-Github-Event') != 'push':
        raise InvalidUsage('Invalid event.')

    assert request.json['repository'].get('full_name') == GITHUB_REPOSITORY
    if request.json['ref'] == 'refs/heads/{}'.format(GITHUB_BRANCH):
        # Triggers deploy :
        try:
            # Updates the repository
            assert subprocess.call('git pull origin prod', shell=True, cwd=ROOT_PATH, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL) == 0, "Unable to update the repository"

            # Update the requirements
            assert subprocess.call('../env/bin/pip install --upgrade -r requirements.txt', shell=True, cwd=ROOT_PATH, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL) == 0, "Unable to update the packages (pip)"

            # Update the database
            assert subprocess.call('../env/bin/python manage.py db upgrade', shell=True, cwd=ROOT_PATH, stderr=subprocess.DEVNULL, stdout=subprocess.DEVNULL) == 0, "Unable to upgrade the database"

            # Touch the reload file to make it restart
            os.utime(os.path.join(ROOT_PATH, 'reload'), None)
        except AssertionError as e:
            raise ServerError(str(e))

    return text("Deployment done.")


if __name__ == '__main__':
    app.run(host='127.0.0.1', port=9001, auto_reload=True, debug=False)

[ahopkins]

Looks super interesting. Will look more closely later, but I’m wondering if this is something we can support natively.

Right now adding a directory for reload also triggers on the regular reload. There is not support for ONLY those dirs.

Development

[cx42net]

Thank you! Feel free to bring any suggestions or better way to handle this via what is actually in the Sanic code.

I tried to look into what was already existing and that’s the best I found, but maybe I missed something critical.

I noticed an issue from my above code, doing this:

auto_reloader = AutoReload(params['auto_reload'])
while auto_reloader.run():
    # Running server
    app.run(**params)

works great, but the app is not regenerated. This means that if, for instance, you change some config parameters (in a .env file for instance), it won’t be taken into consideration.

What is needed instead is a full re-creation of the app inside that while.

The config example is just a case, but there are many other related things an initiating instance can do (connecting to the database => if the config has changed, just calling update on the config object won’t update the actual database connection and will fail later on, same for any other connections (redis, aws, etc)).

Another point to keep in mind is to avoid doing the following:

while AutoReload(params['auto_reload']).run():
    # Running server
    app.run(**params)

This will create a new instance of AutoReload at each loop, and will duplicate the watchdog instances. After three loop, one modification of the “reload” will trigger three “on_modified” events.
For this reason, I’m not a big fan of the loop and the way it is currently handled. Maybe a singleton pattern would be better here to ensure only one instance is running.

[cx42net]

Here’s an update to address my previous concerns:

from sanic.log import logger
from watchdog.events import FileSystemEventHandler
from watchdog.observers import Observer
import os, signal


class ModificationEventLoader(FileSystemEventHandler):
    def __init__(self, handler):
        self.handler = handler

    def on_modified(self, event):
        if event.is_directory:
            return

        self.handler.restart()


class AutoReload:
    """Watches the "reload" file for any changes and triggers a reload of Sanic"""

    _instance = None

    def __init__(self, root_path, auto_reload):
        self.first_loop = True
        self.is_restarting = False
        self.observer = None

        if auto_reload:
            # We print here because the app was not instantiated!
            print('Auto reload is already enabled, no need to setup the watchdog')
            return

        reload_file_path = os.path.join(root_path, 'reload')
        if not os.path.isfile(reload_file_path):
            # We print here because the app was not instantiated!
            print('Missing reload file. Auto reload will be disabled')
            return None

        self.observer = Observer()
        self.observer.schedule(ModificationEventLoader(self), reload_file_path, recursive=False)
        self.observer.start()

    def restart(self):
        """Sets the restart state to true to allow the loop to run again"""
        self.is_restarting = True
        os.kill(os.getpid(), signal.SIGINT)

    def stop(self):
        """Stops the observer, if initiated"""
        if self.observer:
            self.observer.stop()
            self.observer.join()

    @classmethod
    def is_running(cls, root_path, auto_reload):
        """Returns true only at first loop and when a reload is triggered"""
        if not cls._instance:
            cls._instance = cls(root_path, auto_reload)

        if cls._instance.first_loop:
            cls._instance.first_loop = False
            return True

        if cls._instance.is_restarting:
            cls._instance.is_restarting = False
            return True

        cls._instance.stop()

        return False

And this works by calling the following:

    # Setting up watchdog
    while AutoReload.is_running(os.path.abspath(os.path.dirname(__file__)), params['auto_reload']):
        # We need to create the App fully again, to ensure that the initial changes have been made
        # Like ".env" file changes, Database connection, etc

        # In my case, I use a Factory pattern, and retrieve the app
        app = App().get_app()
        app.run(**params)
        app.stop()

        # `app.stop()` should re-register app.name but it's currently not the case (ticket opened)
        # So in the meantime, we do it manually
        # Without this, restarting will fail as the app name is already registered.
        del Sanic._app_registry[app.name]

        app = None

Hope this helps!