Mysterious OverflowError with sanic_session

I’ve encountered these strange OverflowErrors intermittently — but I have trouble finding out what is causing it since I can’t very easily find out what is inside the session objects (if you know how might do that, let me know!)

It seems to me that something inside the session is causing Overflows, but there is only dicts and lists inside the sessions so I am not sure what could cause this.

Traceback (most recent call last):
  File "/path/to/lib/python3.7/site-packages/sanic/app.py", line 977, in handle_request
    request, response, request_name=name
  File "/path/to/lib/python3.7/site-packages/spf/framework.py", line 686, in _run_response_middleware_19_12
    _response = await _response
  File "/path/to/lib/python3.7/site-packages/sanic_session/__init__.py", line 41, in save_session
    await self.interface.save(request, response)
  File "/path/to/lib/python3.7/site-packages/sanic_session/base.py", line 156, in save
    val = ujson.dumps(dict(req[self.session_name]))
OverflowError: Maximum recursion level reached

Any help would be appreciated!

Versions:

sanic==20.6.3
sanic-session==0.7.3

Please, could you add a code snippet? after that, we can figure out what’s going on

I would if I could. Partially, the reason that I wasn’t able to figure out what’s wrong is:

  • traceback doesn’t contain any part of my code
  • I don’t know the URL it gets called from. For example, our app uses New Relic but there are no logs on the New Relic that references this.
  • I have a custom log that supposedly logs everything that the logger sanic.root does, plus some of my internal messages. I don’t see logs on that log either.

So I am somewhat lost as to how I might determine the condition that produces this. And that’s why I am asking here to look for possible strategies to do just that!

Something went wrong in your handler’s code.

when you put a recursive dict into session, It happens.

try this:

from sanic import Sanic
from sanic.response import text
from sanic_session import Session, InMemorySessionInterface

app = Sanic()

session = Session(app, interface=InMemorySessionInterface())


@app.route("/")
async def index(request):
    a = b = dict()
    a["b"] = b
    request.ctx.session["foo"] = a
    return text(123)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)
Traceback (most recent call last):
  File "/Users/zinklu2/.virtualenvs/sanic/lib/python3.7/site-packages/sanic/app.py", line 977, in handle_request
    request, response, request_name=name
  File "/Users/zinklu2/.virtualenvs/sanic/lib/python3.7/site-packages/sanic/app.py", line 1281, in _run_response_middleware
    _response = await _response
  File "/Users/zinklu2/.virtualenvs/sanic/lib/python3.7/site-packages/sanic_session/__init__.py", line 41, in save_session
    await self.interface.save(request, response)
  File "/Users/zinklu2/.virtualenvs/sanic/lib/python3.7/site-packages/sanic_session/base.py", line 156, in save
    val = ujson.dumps(dict(req[self.session_name]))
OverflowError: Maximum recursion level reached

It’s legal to make a recursive data struct in python, But when it dumps into json string, the OverflowError will be raised.

That why the traceback information showed exceptions in sanic_session's code rather than your handler’s code.

2 Likes

I see. I have looked into it and I think that I may have found the culprit. I would still have to figure out what recursive data is there, but right now I am wondering if it had to do with python-box https://pypi.org/project/python-box/

The session interface implies that I can put anything inside but maybe I should only put whatever json allows? I tried to debug this issue by converting everything to dicts before assigning to sessions — but if I do that then it’s hard to work with the objects since they now become dicts.

In any case though, this is very helpful!

What would you suggest for me to try to handle this? Should I do something like:

try:
   request.ctx.session["foo"] = a
except Exception as e:
   logger.exception(e)

so I can determine what is causing this?

Well, I would do a data verification to avoid recursive data before put into session, cause It seems serialize recursive data may waste more server resource.

Or catch a global exception is a better idea;

@app.exception
def except_overflow(request, error: OverflowError):
      ...

Thouhgh I don’t know you code

try:
   request.ctx.session["foo"] = a
except Exception as e:
   logger.exception(e)

is also a great choice

Thank you. I have found the culprit — and incredibly it doesn’t even have to do with session value assignments. I store the language of the session inside the session object, and it seems that some sessions were tainted with an impossible key, and that key in turn made Babel throw an error (sanic-babel) — but that Error was not caught by anything and so it caused the OverflowError on the session.

This is one of the issues I have found that is hard to debug. Fact is, that exception cannot actually be caught by your suggested code here:

@app.exception
def except_overflow(request, error: OverflowError):
      ...

I have already had a catch all top level exception catching which is like this:

def handle_top_level_exception(request, exception):
  pass

app.error_handler.add(Exception, handle_top_level_exception)

and it didn’t catch any of those errors at all. There were a few instances during development with exceptions that some errors cannot be caught using this method of catching all errors. I can’t think of examples now but I know that it exists and they always took a long while for me to debug.

Anyway, thank you very much!

1 Like

Sorry, The right snippet should be

@app.exception(OverflowError)
def except_overflow(request, error: OverflowError):
      ...

but it tried and found that neither

@app.exception(OverflowError)
def except_overflow(request, error: OverflowError):
      ...

nor

try:
   request.ctx.session["foo"] = a
except Exception as e:
   logger.exception(e)

could catch the exception.

request.ctx.session["foo"] = a also won’t raise any exception cause the json serialization happens in a middleware handler.

And app.exception can’t catch because sanic handle middleware exception automatically, it won’t be handle by app.error_handler.

But since you found the problem that would be easy to fix.

1 Like

Sorry for the irresponsible answer.

I think that you have identified the issue:

And app.exception can’t catch because sanic handle middleware exception automatically, it won’t be handle by app.error_handler .

So right now is there anyway I can catch those errors? I’ve had no issues catching errors if I wrote my own middleware. But in all of the problematic cases, those errors were thrown by third-party libraries and I don’t want to edit their code because it makes it hard to maintain.

Is it possible to catch middleware exceptions?

Here is sanic handle_request method code snippet

Sanic do catch request_middleware and hanlder exceptions. but hanlde response_middleware exception by itself (line 976:988)

It seems there is no way to catch response_middleware exceptions so far.

Is there any chance to use pickle.dumps rather than ujson.dumps in sanic session?

Is there any chance to use pickle.dumps rather than ujson.dumps in sanic session?

Unfortunately, no. The dump is written directly inside the library, and I can’t assign a different serializer — to do that I would have to fork that repo and write my own. I’ve already explained why I try not to do that (hard to maintain if I have custom forks of all the libraries I use) but I can probably open an issue and see if the maintainer would consider adding the option to use a custom serializer.

I’ve faced this exact issue in aiocache but they do support custom serializer and that solved all my problems.

1 Like