How to ensure a future is only run once even with multiple workers

Questions and Help
#1

I have different services like a database and a download service. The download service runs(once) on startup and downloads data (from an external source).

The problem is:
when I have workers set to more than one sanic will run after_server_start for every worker and also run all task submitted by app.add_task the same amount as workers (this even is true if the task is not added in after_server_start)

This behavior is fine for my database but I do not want the download service to run multiple times how can i stop that ?

sidenote:
i tried manual locking the service (make a bool if it is true don’t start the download process) but for some reason there seems to be no impact (maybe due to some multi processing shenanigans) also this solution seems hideous

here some test code with the output:

from sanic import Sanic
from sanic.response import json

app = Sanic()
execution_count = 0
async def this_function_should_only_run_in_one_thread():
    global execution_count
    execution_count += 1
    print("Executed")
    
async def startup(app, loop):
    app.add_task(this_function_should_only_run_in_one_thread())
    

app.register_listener(startup, 'after_server_start')


@app.route("/")
async def test(request):
    return json({"executed": execution_count})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=1337,workers=4)

this will output

806×210 4.68 KB

#2

Hello, @K0IN!

Well, this is a normal problem when you deal with workers > 1: the multiprocessing module will spawn (or fork) everything prior to starting Sanic, so your application will always execute that function for the same number of workers you have.

One solution would be to call your function prior to calling app.run and, if you want to share something from that function with the app, you could easily do so by just adding a variable to app (this will only work if you don’t plan on modifying that value, let’s say, a file path for a GeoIP database you downloaded earlier).

In case you need to share a state between the Sanic workers, then you’ll probably need a different approach, like a pubsub mechanism.

I hope this helps! :smile: