How to best-manage long-running / expensive requests?

#1

Hey there,

I need to process a request, which runs an expensive CPU-bound process based on a redis request.

The process is simply-parallelizable, such that I could normally use a multiprocessing Pool, but unfortunately those can’t be called from a Sanic request function - as the Sanic workers are already running in multiprocessing workers.

What’s the best approach for handing long-running CPU bound tasks? thanks!

#2

Usually the answer is to farm it out to another process where you can. There are a number of task managers in Python.

Here is a talk I gave about background processing in Sanic. At the end there is a discussion of Sanic based background solution I created using multiprocessing queues.

LMK if you have any other specific questions.

1 Like