Using asyncio.subprocess on Windows fails with Sanic

Questions and Help
#1 
import asyncio
from sanic import Sanic
from sanic.response import empty

app = Sanic(__name__)

@app.get("/")
async def unused(request):
    return empty()

@app.before_server_start
async def run(*ignored):
    proc = await asyncio.subprocess.create_subprocess_exec("echo", "foo")
    await proc.wait()

if __name__ == "__main__":
    asyncio.run(run())

If the above is run as script, it prints “foo”, as is expected (but exits without running Sanic). When ran with Sanic CLI (or if code is modified to use app.run), I get

  File "mycode.py", line 13, in run
    proc = await asyncio.subprocess.create_subprocess_exec("echo", "foo")
  File "C:\Python311\Lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "C:\Python311\Lib\asyncio\base_events.py", line 1680, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "C:\Python311\Lib\asyncio\base_events.py", line 502, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError

Apparently Sanic does something with asyncio loop/internals to make asyncio.subprocess NotImplemented. Running the same code on Linux works fine. I used before_server_start but the same happens anywhere within a Sanic app (handlers etc). The same also on Python 3.10 (instead of 3.11 used here).

#2

It would seem that I need ProactorEventLoop on Windows, and that Python 3.8+ has made that the default but that sanic/server/loop.py switches to SelectorEventLoop instead. Unfortunately the code doesn’t have any comments as to why.

This was written/touched by @ahopkins four months ago as a part of the WorkerManager refactoring. Is there some reason why ProactorEventLoop doesn’t work with Sanic, or why the change?

#3

FWIW, simply changing to Proactor seems to be working, even with --fast although then I get nasty errors like

Accept failed on a socket
socket: <asyncio.TransportSocket fd=744, family=2, type=1, proto=6, laddr=('127.0.0.1', 8000)>
Traceback (most recent call last):
  File "C:\Python311\Lib\asyncio\proactor_events.py", line 856, in loop
    f = self._proactor.accept(sock)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\asyncio\windows_events.py", line 574, in accept
    self._register_with_iocp(listener)
  File "C:\Python311\Lib\asyncio\windows_events.py", line 743, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] The parameter is incorrect

during server startup.

#4

Yeah, I should have wrote some notes on it. I did a bunch of research, and that was the recommendation most often given. I’d suggest other patterns to push subprocess calls async, like a queue to a managed process that can then run with regular subprocess.

1 Like
#5

I’d prefer Proactor, though, as it appears to have a number of benefits anyway and is recommended by Python and Microsoft now. Any idea what specifically breaks in Sanic with it? Guessing from that error message I assume the fileno is invalid after transferred to worker process but somehow it was still serving my requests.