byte constants are not decompiled correctly

[mristin]

Hi,

I've just encountered a serious bug with Python 3. Namely, the byte constants are not decompiled correctly.

The string constants are decompiled correctly:

>>> import ast
>>> import meta
>>> import meta.decompiler

>>> f=lambda: 'oi'
>>> tree = meta.decompiler.decompile_func(f)
>>> ast.dump(tree)
"Lambda(args=arguments(args=[], vararg=None, kwonlyargs=[], kw_defaults=[], kwarg=None, defaults=[]), body=Return(value=Str(s='oi')))"

However, when I run the similar example with the bytes:

>>> import ast
>>> import meta
>>> import meta.decompiler

>>> f=lambda: b'oi'
>>> tree = meta.decompiler.decompile_func(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/__init__.py", line 37, in decompile_func
    ast_node = make_function(code, defaults=[], lineno=code.co_firstlineno)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 137, in make_function
    stmnts = instructions.stmnt()
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 310, in stmnt
    self.visit(instr)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/instructions.py", line 324, in visit
    method(instr)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/simple_instructions.py", line 279, in RETURN_VALUE
    value = self.process_ifexpr(value)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/simple_instructions.py", line 343, in process_ifexpr
    return ExpressionMutator().visit(node)
  File "/usr/lib/python3.5/ast.py", line 245, in visit
    return visitor(node)
  File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/expression_mutator.py", line 39, in generic_visit
    return NodeTransformer.generic_visit(self, node)
  File "/usr/lib/python3.5/ast.py", line 295, in generic_visit
    for field, old_value in iter_fields(node):
  File "/usr/lib/python3.5/ast.py", line 170, in iter_fields
    for field in node._fields:
AttributeError: 'bytes' object has no attribute '_fields'

I see that at File "/home/marko/workspace/pqryopen/icontract/venv3/lib/python3.5/site-packages/meta/decompiler/expression_mutator.py", line 39, in generic_visit you don't check for bytes, only for strings:

class ExpressionMutator(NodeTransformer):
    def visit_If(self, node):

        assert len(node.body) == 1

        assert len(node.orelse) == 1

        test = self.visit(node.test)
        then = self.visit(node.body[0])
        else_ = self.visit(node.orelse[0])

        if_exp = _ast.IfExp(test, then, else_, lineno=node.lineno, col_offset=0)
        return if_exp

    def visit_Return(self, node):
        return NodeTransformer.generic_visit(self, node.value)

    def visit_FunctionDef(self, node):
        return node

    def generic_visit(self, node):
        if node is None:
            return node
        if isinstance(node, (str)):
            import pdb;pdb.set_trace()
            return node

#        if not isinstance(node, (_ast.expr, _ast.expr_context, _ast.slice, _ast.operator, _ast.boolop)):
#            raise Exception("expected a Python '_ast.expr' node (got %r)" % (type(node),))
        return NodeTransformer.generic_visit(self, node)

This is a bit of a blocker for us since we need meta in our library icontract to impose contract validations (pre- and post-conditions).

I tried changing the generic_visit in expression mutator to:

    def generic_visit(self, node):
        if node is None:
            return node
        if isinstance(node, str):
            return _ast.Str(node)
        if isinstance(node, bytes):
            return _ast.Bytes(node)

        return NodeTransformer.generic_visit(self, node)

And that seems to work.

I'd like to fix the issue and make a pull request, but before I go forward with it, could you please explain me:

• Why is pdb locally imported followed by a call pdb.set_trace()?
• Why are strings returned as strings and not as _ast.Str?

Thaks for looking into this!

[mristin]

Hi @srossross , I've just started with this issue. However, I'm quite confused with the compatibility between Python 2 and 3 and how Meta was meant to resolve it. Namely, when I write a unit test in meta/decompiler/tests/test_simple.py:

    @py3only
    def test_byte_constant(self):
        stmnt = 'b""'
        self.statement(stmnt)

and fix the code, the test fails with:

AssertionError: Ast Not Equal:
Generated: "Module(body=[Expr(value=b'')])\n"
Expected:  "Module(body=[Expr(value=Bytes(s=b''))])\n"

If I am correct, this is due to Python 2 versus Python 3 handling of bytes? Is the test suite intended to generate expressions of Python 2 or Python 3?

Thanks a lot for all your help!