How to populate context in main with sanic 22.12

[jschewebbn]

I want to put data from the command line into app.ctx so that my workers have access to it. Using version 22.9.1 this works using 22.12 the property isn’t set when accessed in the workers.

Sample application

#!/usr/bin/env python3

from sanic import Sanic
from sanic import response as res

app = Sanic(__name__)

@app.route("/")
async def test(req):
    return res.text("I\'m a teapot", status=418)


@app.before_server_start
def worker_init(app):
    print("Argument is ", app.ctx.arg1)


if __name__ == '__main__':
    app.ctx.arg1 = 'argument from command line'
    app.run(host="0.0.0.0", port=8000)

This works in 22.9.1, however 22.12 results in the following error.

[2023-01-19 14:45:09 -0600] [68062] [ERROR] 'types.SimpleNamespace' object has no attribute 'arg1'
Traceback (most recent call last):
  File "/home/jschewe/projects/sanic-bugs/pass-command-line/venv/lib/python3.10/site-packages/sanic/worker/serve.py", line 117, in worker_serve
    return _serve_http_1(
  File "/home/jschewe/projects/sanic-bugs/pass-command-line/venv/lib/python3.10/site-packages/sanic/server/runners.py", line 231, in _serve_http_1
    loop.run_until_complete(app._server_event("init", "before"))
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/home/jschewe/projects/sanic-bugs/pass-command-line/venv/lib/python3.10/site-packages/sanic/app.py", line 1598, in _server_event
    await self.dispatch(
  File "/home/jschewe/projects/sanic-bugs/pass-command-line/venv/lib/python3.10/site-packages/sanic/signals.py", line 195, in dispatch
    return await dispatch
  File "/home/jschewe/projects/sanic-bugs/pass-command-line/venv/lib/python3.10/site-packages/sanic/signals.py", line 165, in _dispatch
    retval = await maybe_coroutine
  File "/home/jschewe/projects/sanic-bugs/pass-command-line/venv/lib/python3.10/site-packages/sanic/app.py", line 1177, in _listener
    maybe_coro = listener(app)  # type: ignore
  File "/home/jschewe/projects/sanic-bugs/pass-command-line/simple_app.py", line 15, in worker_init
    print("Argument is ", app.ctx.arg1)
AttributeError: 'types.SimpleNamespace' object has no attribute 'arg1'

I expect this is related to the worker manager rewrite.

[ahopkins]

it’s because you’re adding to ctx I main only. You need to do it in global or in a factory. If you can, the recommendation is to use the cli, which does allow you to add arbitrary flags. Or, of course env vars.

[jschewebbn]

Given that the information comes from the command line, I expect that I must gather that information in main rather than globally. Can you point me to an example that fits “the recommendation is to use the cli”?

[ahopkins]

I need more context and more of a code snippet of what you are trying to do.

But you can try using shared_ctx

Worker Manager

[jschewebbn]

I want to take the value of a command line argument and make it available to all workers as configuration information. Here’s more detail, although I expect you can gather this from the above example. Perhaps the key for you is that the command line argument is not something specific to sanic.

#!/usr/bin/env python3

from sanic import Sanic
from sanic import response as res
import sys

app = Sanic(__name__)

@app.route("/")
async def test(req):
    return res.text("I\'m a teapot", status=418)


@app.before_server_start
def worker_init(app):
    print("Argument is ", app.ctx.arg1)
    # use app.ctx.arg1 to configure some aspect of the worker such as it's logging configuration


if __name__ == '__main__':
    app.ctx.arg1 = sys.argv[1]
    app.run(host="0.0.0.0", port=8000)

[jschewebbn]

I spent more time with shared_ctx today and got it working. For others that want to do this here is the example.

#!/usr/bin/env python3

from sanic import Sanic
from sanic import response as res
import sys

app = Sanic(__name__)

@app.route("/")
async def test(req):
    return res.text("I\'m a teapot", status=418)


@app.before_server_start
async def worker_init(app):
    print("worker: shared_ctx", app.shared_ctx)
    print("worker: arg1", app.shared_ctx.arg1)

    
@app.main_process_start
async def main_process(app):
    print("In main process start")
    print("main process: shared_ctx", app.shared_ctx)
    

if __name__ == '__main__':
    app.shared_ctx.arg1 = sys.argv[1]
    app.run(host="0.0.0.0", port=8000)
    

Documented at Worker Manager | Sanic Framework per the post from @ahopkins. Note that only simple objects can be put in shared_ctx.

[jschewebbn]

Not quite a solution, it appears that a string is not a simple object.

[jschewebbn]

Current attempt is this, but sanic hangs

#!/usr/bin/env python3

from sanic import Sanic
from sanic import response as res
import sys
import multiprocessing
import ctypes

app = Sanic(__name__)

@app.route("/")
async def test(req):
    return res.text("I\'m a teapot", status=418)


@app.before_server_start
async def worker_init(app):
    arg1 = app.shared_ctx.arg1.value
    print("worker: arg1", arg1)
    print("After")

    
@app.main_process_start
async def main_process(app):
    print("In main process start")
    app.shared_ctx.arg1 = multiprocessing.Value(ctypes.c_wchar_p, sys.argv[1])
    arg1 = app.shared_ctx.arg1.value
    print("main process: arg1", arg1)
    

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8000)   

Output

[2023-01-27 16:51:08 -0600] [77029] [WARNING] Sanic is running in PRODUCTION mode. Consider using '--debug' or '--dev' while actively developing your application.
In main process start
main process: arg1 cli
[2023-01-27 16:51:39 -0600] [77029] [ERROR] Not all workers acknowledged a successful startup. Shutting down.

It seems that one or more of your workers failed to come online in the allowed time. Sanic is shutting down to avoid a deadlock. The current threshold is 30.0s. If this problem persists, please check out the documentation https://sanic.dev/en/guide/deployment/manager.html#worker-ack.
[2023-01-27 16:51:39 -0600] [77029] [INFO] Killing Sanic-Server-0-0 [77038]
[2023-01-27 16:51:39 -0600] [77029] [INFO] Server Stopped

The examples from multiprocessing — Process-based parallelism — Python 3.11.1 documentation work fine, but once I try and use multiprocessing.Value inside Sanic, the application hangs in the worker process.

[jschewebbn]

Figured it out. I need to use a RawArray type otherwise something in the Sanic SharedContext implementation causes a deadlock. For my purposes this is OK as the values that I’m setting are read-only.

#!/usr/bin/env python3

import sanic.types
from sanic import Sanic
from sanic import response as res
import sys
import multiprocessing
import ctypes
import json
import logging
from pathlib import Path
import os
import pickle
import argparse
from typing import Any

app = Sanic(__name__)

def get_logger():
    return logging.getLogger(__name__)


@app.route("/")
async def test(req):
    get_logger().debug("debug message")
    return res.text(f"I\'m a teapot", status=418)


@app.before_server_start
async def worker_init(app):
    args = pickle.loads(app.shared_ctx.args)
    setup_logging(default_path=args.logconfig)


def setup_logging(
    default_path='logging.json',
    default_level=logging.INFO,
    env_key='LOG_CFG'
):
    """
    Setup logging configuration
    """
    try:
        path = Path(default_path)
        value = os.getenv(env_key, None)
        if value:
            path = Path(value)
        if path.exists():
            with open(path, 'r') as f:
                config = json.load(f)
            logging.config.dictConfig(config)
        else:
            logging.basicConfig(level=default_level)
    except:
        print(f"Error configuring logging, using default configuration with level {default_level}")
        logging.basicConfig(level=default_level)

        
if __name__ == '__main__':
    class ArgumentParserWithDefaults(argparse.ArgumentParser):
        """
        From https://stackoverflow.com/questions/12151306/argparse-way-to-include-default-values-in-help
        """
        def add_argument(self, *args, help=None, default=None, **kwargs):
            if help is not None:
                kwargs['help'] = help
            if default is not None and args[0] != '-h':
                kwargs['default'] = default
                if help is not None:
                    kwargs['help'] += ' (default: {})'.format(default)
            super().add_argument(*args, **kwargs)
        
    parser = ArgumentParserWithDefaults(formatter_class=argparse.RawTextHelpFormatter)
    parser.add_argument("-l", "--logconfig", dest="logconfig", help="logging configuration (default: logging.json)", default='logging.json')

    args = parser.parse_args(sys.argv[1:])
    app.shared_ctx.args = multiprocessing.RawArray('c', pickle.dumps(args))
    app.shared_ctx.args = args
    
    app.run(host="0.0.0.0", port=8000)