List comprehension not decompiled

[mristin]

Hi, Similar to issue #41, the list comprehension is not decompiled:

>>> import meta.decompiler
>>> meta.decompiler.decompile_func(lambda x: [i for i in range(x)])

This gives me a traceback:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/__init__.py", line 37, in decompile_func
    ast_node = make_function(code, defaults=[], lineno=code.co_firstlineno)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 137, in make_function
    stmnts = instructions.stmnt()
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 310, in stmnt
    self.visit(instr)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 324, in visit
    method(instr)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 372, in MAKE_FUNCTION
    function = make_function(code, defaults, lineno=instr.lineno, annotations=annotations, kw_defaults=kw_defaults)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 135, in make_function
    instructions = Instructions(disassemble(code))
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/disassemble.py", line 20, in disassemble
    return list(disassembler(co))
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/bytecodetools/disassembler_.py", line 25, in disassembler
    code = co.co_code
AttributeError: 'Str' object has no attribute 'co_code'

I suppose the solution to #41 will be the same as here?

[mristin]

Hi @srossross , Any chance you could have a look at this issue the next week? It is blocking us to write predicates in our contracts (https://github.com/Parquery/icontract) and they are so omnipresent that it is hard to re-write them in a different way.

I'd really like to give you a hand with these issues. Please let me know how I could help -- maybe we could schedule a skype meeting with a shared screen and look at it together for a while (max 30min) and I would try to figure out how to fix the problem in more depth afterwards on my own?

I would be available for a call from Sept 8th onward.

[mristin]

The offending object is on meta/bytecodetools/disassembler_.py:29:

def disassembler(co, lasti= -1):
    """Disassemble a code object. 
    
    :param co: code object
    :param lasti: internal
    :yields: Instructions.
    
    """
    print('''co is {}'''.format(co))  # TODO: debug remove
    if hasattr(co, 's'):
        print('''co.s is {}'''.format(co.s))  # TODO: debug remove

    code = co.co_code

In the offending case, the output is:

co is <_ast.Str object at 0x7fe9e3fdb4a8>
co.s is <lambda>.<locals>.<genexpr>

[mristin]

Here is the trace of some values as they pass through the functions:

in meta.decompiler.instructions.make_function: code is <code object <lambda> at 0x7fbd38ed6780, file "/home/marko/.PyCharmCE2018.1/config/scratches/scratch.py", line 2>
in meta.decompiler.disassemble.disassemble: co is <code object <lambda> at 0x7fbd38ed6780, file "/home/marko/.PyCharmCE2018.1/config/scratches/scratch.py", line 2>
in meta.bytecodetools.disassembler_: co is <code object <lambda> at 0x7fbd38ed6780, file "/home/marko/.PyCharmCE2018.1/config/scratches/scratch.py", line 2>
in meta.decompiler.instructions.visit: instr is   2           0 LOAD_CONST               1 (<code object <listcomp> at 0x7fbd3a4ab9c0, file "/home/marko/.PyCharmCE2018.1/config/scratches/scratch.py", line 2>)
in meta.decompiler.instructions.visit: instr is               3 LOAD_CONST               2 ('<lambda>.<locals>.<listcomp>')
in meta.decompiler.instructions.visit: instr is               6 MAKE_FUNCTION            0
in meta.decompiler.instructions.MAKE_FUNCTION: code is <_ast.Str object at 0x7fbd38e60cc0>
in meta.decompiler.instructions.make_function: code is <_ast.Str object at 0x7fbd38e60cc0>
in meta.decompiler.disassemble.disassemble: co is <_ast.Str object at 0x7fbd38e60cc0>
in meta.bytecodetools.disassembler_: co is <_ast.Str object at 0x7fbd38e60cc0>
co.s is <lambda>.<locals>.<listcomp>

The content of the scratch file is:

import meta.decompiler
meta.decompiler.decompile_func(lambda x: [i for i in range(x)])

[mristin]

Hi @srossross , Is there any chance that you might find time to look into this with me this week?

I don't expect it to take more than 30 minutes, I would just like to discuss with you what the exact problem is (and whether there are bigger repercussions which I am unaware of) before starting to work on a pull request.

Please also let me know if you are busy and won't have any spare time soon so that we dive into the code without any previous discussion and hack out the pull request.

[srossross]

Hi @mristin, my time to work on this in the coming months will be very tight. It would be great if you could create a pull request.

[mristin]

Hi @srossross ,

Thanks for letting me know. I've just started with the issue #39 .

[mristin]

Hi @srossross , I tried hard with #39, but realized on the way that updating meta library for our purposes is actually a cause perdu. We would always have to keep up with new Python versions and this is much of an overkill for our application (getting the source code from a lambda function).

In the end, I decided to remove dependency on meta from icontract library (https://github.com/Parquery/icontract/commit/0e3edf5194fee9609d0b32e56ece7b288421570b) and decided to reparse the whole source code from the file. This approach, while way less efficient than using meta, works for our use cases and hopefully needs fewer updates with new Python versions.

I'll leave the issues open in case somebody wants to jump in and fix them.