Empty stacktrace

[JulianOrteil]

Maybe I missed something when reading through your README, but I am getting an empty stack trace.

I followed your example in your README, but this is what I see. I tried previous versions (0.2.0, 0.1.1), but got an empty trace as well. I tried this on both Python 3.7.7 and Python 3.8.5 with no change.

[alexmojaki]

The source code isn't available in the shell, which is why standard python tracebacks are also empty. Use it in a script.

[JulianOrteil]

I see. It might be useful to mention that in the README.

Thank you.

[cknd]

(It can still be useful to have it enabled in interactive sessions, because exceptions in code called from the shell will work. Alternatively, in ipython this example also works)

[JulianOrteil]

@cknd I do a lot of my test-code in the shell and I'm sure I'm not the only one; so I'll leave it to your discretion if this something you should put effort into. In my experience, the more versatile a library like this is, the better experience users have.

[cknd]

Of course, it would be nice! The trouble is that the plain python shell doesn't allow me to access the source code (or at least, the inspect module doesn't find it), so I don't know where to start on this one.

I do most of my code testing in ipython (mostly because of other conveniences like better tab completion), maybe that's a workaround

[JulianOrteil]

Any reason why you can't use the 'dill' library?

That works perfectly fine in the standard shell for getting source code.

[JulianOrteil]

Users will need to install both 'dill' and 'pyreadline' from pip. Here's the specific function that can be used if you just want to port their implementation over to stackprinter:

https://github.com/uqfoundation/dill/blob/5c81867f2a9edceb3ab7d3bbac83a7f422c9d7a1/dill/source.py#L330

[cknd]

Thanks!

[alexmojaki]

@JulianOrteil that's very interesting! However using (py)readline gives you the full interpreter history as a source 'file', which can be good enough if you want the source code of a single function or class, but dill doesn't seem to know what to do with a frame or code object:

>>> dill.source.getsource(inspect.currentframe())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 354, in getsource
    lines, lnum = getsourcelines(object, enclosing=enclosing)
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 325, in getsourcelines
    code, n = getblocks(object, lstrip=lstrip, enclosing=enclosing, locate=True)
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 251, in getblocks
    lines, lnum = findsource(object)
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 154, in findsource
    raise IOError('could not extract source code')
OSError: could not extract source code
>>> def foo(): pass
...
>>> dill.source.getsource(foo.__code__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 354, in getsource
    lines, lnum = getsourcelines(object, enclosing=enclosing)
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 325, in getsourcelines
    code, n = getblocks(object, lstrip=lstrip, enclosing=enclosing, locate=True)
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 251, in getblocks
    lines, lnum = findsource(object)
  File "/home/alex/.cache/pypoetry/virtualenvs/book-E12DIahb-py3.8/lib/python3.8/site-packages/dill/source.py", line 154, in findsource
    raise IOError('could not extract source code')
OSError: could not extract source code
>>> dill.source.getsource(foo)
'def foo(): pass\n'

Plus this is getting real hacky in other ways. For example pyreadline hasn't had a commit in 5 years and readline isn't always available even on platforms that should have it.

[JulianOrteil]

@alexmojaki

I looked into this a bit further, like you have, and agree. That is hacky. I provided dill as a pointer to see if it would help get the ball rolling. Based on that, I did come across trace hooking. Just using the sys module, you should be able to watch every call of a program, in any thread (https://docs.python.org/3.8/library/sys.html#sys.settrace), using 'sys.settrace'. It seems to be cross-platform and works in the interactive shell and files. If you take a look at that docs link, take note of the CPython warning at the bottom.

You should also be able to get the arguments, returns, and exceptions from every function as 'settrace' passes the frame-object to your handler (like what 'sys.excepthook' does).

I have provided my testing code below:

# Trace all function calls of a program and only catch exceptions
import sys

def trace_exceptions(frame, event, arg):
    if event != 'exception':
        return
    co = frame.f_code
    func_name = co.co_name
    line_no = frame.f_lineno
    filename = co.co_filename
    exc_type, exc_value, exc_traceback = arg
    print('Tracing exception: %s "%s" on line %s of %s' % (exc_type.__name__, exc_value, line_no, func_name))

def trace_calls(frame, event, arg):
    if event != 'call':
        return
    co = frame.f_code
    func_name = co.co_name
    if func_name in TRACE_INTO: # Optional line, just for testing purposes
        return trace_exceptions

def c():
    raise RuntimeError('generating exception in c()')

def b():
    c()
    print('Leaving b()')

def a():
    b()
    print('Leaving a()')

TRACE_INTO = ['a', 'b', 'c']

sys.settrace(trace_calls)
try:
    a()
except Exception as e:
    print('Exception handler:', e)

And the code for tracking function calls, their returns, and args. I've left it broken out so you can review it easily.

# Track the calls and returns of functions
import sys

def trace_calls_and_returns(frame, event, arg):
    co = frame.f_code
    func_name = co.co_name
    if func_name == 'write':
        # Ignore write() calls from print statements
        return
    line_no = frame.f_lineno
    filename = co.co_filename
    if event == 'call':
        print('Call to %s on line %s of %s' % (func_name, line_no, filename))
        return trace_calls_and_returns
    elif event == 'return':
        print('%s => %s' % (func_name, arg))
    return

def b():
    print('in b()')
    return 'response_from_b '

def a():
    print('in a()')
    val = b()
    return val * 2

sys.settrace(trace_calls_and_returns)
a()

# Trace the args of functions
import sys

def trace_args(frame, event, arg):
    if event != 'call': return
    print("Called", frame.f_code.co_name)
    for i in range(frame.f_code.co_argcount):
        name = frame.f_code.co_varnames[i]
        print("\tArgument", name, "is", frame.f_locals[name])

sys.settrace(trace_args)

def kenobi(a, b, c):
    print(a, b, c)

kenobi("Hello", "there", "!")

Stack tracing is still very new to me, so I apologize if this isn't of help.

[alexmojaki]

I'm very familiar with settrace. I don't see how it would help get the source code in the shell. It would also get in the way of other debuggers.

[JulianOrteil]

I'm not proposing settrace as the catch-all, end-all for the printer. I can see the complications of using settrace and its incompatibilities with other utilities. I'm proposing it as a fallback/alternative when running or importing files from the shell.

Neither dill's getsource, nor sys.settrace can be used independently to solve this issue, so could they be used together? We're able to detect if code is running in the shell (or import from a file into the shell) because sys.argv will be empty. With this being said, falling back to my proposed solution (only when in the shell) might be able to help solve the issue. Also, that behavior could be disabled by setting an environment variable before stackprinter is imported to the shell. I've seen many other utilities do that.

For example:

1. Per new documentation of this solution, the user must set an environ variable named 'STACKPRINTER_ENABLE_SETTRACE' to 1 to enable this behavior before stackprinter is imported (this behavior could be enabled by default, but I leave that to your discretion, because as you've stated, it could interfere with other debuggers and utilities).
2. Upon import, stackprinter checks if sys.argv is empty (indicating we're running in the shell) and if 'STACKPRINTER_ENABLE_SETTRACE' is 1. Not being able to find this environ variable indicates the user wishes for this behavior to be disabled. If this behavior is disabled, we should throw out a user warning indicating that stackprinter will not work properly in a shell.
3. Using a custom function, stackprinter should be able to read the source code using a combination of a ported version of dill's getsource and the frame objects provided by settrace.

I understand that its not the perfect solution; however, it's better than leaving an empty stacktrace with no documentation on why that occurs.

[JulianOrteil]

Or heck, you could extend the functionality of 'set_excepthook' by adding another parameter to the function like 'enable_settrace' if you wish to not use an environment variable. I actually think that would be a better way for the user to enable this behavior.

[alexmojaki]

Any exception hook already has full access to all the frames involved. settrace does not provide any new information. What I showed you above is that dill isn't able to get the source code of the frame.

Getting the source code in the shell is:

1. Perhaps impossible
2. Definitely very difficult if possible.
3. Not very useful. Serious coding should happen in scripts. If an interactive shell is important to you, use ipython. The standard python shell exists to quickly try things out with zero setup, the experience and functionality is expected to be poor. I expect that 90% of the time that people use stackprinter in the shell it's to try the library out, not because they really need good stacktraces.

I think a reasonable feature to add would be a message like (could not retrieve source code for some frames) at the bottom whenever this happens for any reason (e.g. if the code only exists in .pyc files) and maybe an additional message (source code cannot be retrieved in the shell, use ipython or a script for proper tracebacks) specifically for the case of the shell, just so that people trying out the library know what's going on.

[JulianOrteil]

Ah, I see where you are now. Sorry for the misunderstanding.

I agree. I think that would be beneficial for people just using the shell.