Multiprocessing alternatives?

Questions and Help
#1

I am building an application that will fetch and process N resources and then return a single aggregate result.

I would like to use concurrent.futures.ProcessPoolExecutor to execute processing on the N resources in parallel, but found that this does not work because Sanic already is managing a process pool (something like that anyway).

So, I had another idea: Create an endpoint: /fetch_and_process_one_datum

I can call that in my main endpoint with:
request.app.asgi_client.get("/fetch_and_process_one_datum?params=...") (but, inside a thread executor) so it will fetch them in parallel, and basically create N new requests. Then it will aggregate the results and return.

Is this an OK way to do this? Is there a better alternative?

it feels a little hacky making an endpoint that is only for internal purposes, and I wonder if there is a better way, but I could not find anything in the docs or issues.

Thanks!

#2

I do not personally like that approach. Yes, it is hacky. The way I see it you have two choices:

  1. Use single_process mode so that you can control multiprocessing
  2. Create a number of “workers” up front as your worker pool and pass to the pool through a shared queue.