This is my simple code (on ubuntu):
async def gen(lis): # A generator for simulating streaming returns for t in lis: time.sleep(0.01) yield t + "\n\n" async def chat_completions(request: Request): request_body = json.loads(request.body) # not used response = await request.respond() lis = ["qweryoiuqwpr" + str(i) for i in range(1000)] g = gen(lis) # A generator for simulating streaming returns async for r in g: await response.send(r) await response.eof()
All response were cached and send only when there is no
response.send() any more.
But when I changed the func gen() to:
async def gen(lis): for t in lis: await asyncio.sleep(0.01) # asyncio.sleep(0) also work. #This method disrupts the continuity of the current event yield t + "\n\n"
It works as expected.
So, it seems that there is a continuous ‘write’ operations when run
await response.send(r), and
asyncio.sleep(0.01) can break it. (
asyncio.sleep(0) can also break it)
Can someone tell me how to resolve this problem with out using
asyncio.sleep(0), since this can lead to frequent CPU overhead which is non negligible.