Pyrlang
Decoding/Encoding issues
Hi!
First of all - thank you for building those two libraries!
That's absolutely amazing to see some activity in this are, and it makes me super excited to be able to build a node that can easily communicate with other BEAM based technologies!
When playing with the code, I've discovered a couple of problems.
Let me start with my Environment/setup - initially, I starting with Elixir, but then, I've double confirmed with "pure" Erlang.
Environment:
elixir --version
Erlang/OTP 21 [erts-10.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
Elixir 1.7.3 (compiled with Erlang/OTP 19)
Setup
Given I have a node implemented with Pyrlang:
from pyrlang import Node, GeventEngine, GenServer
from term import Atom
import pickle
class PyGenServer(GenServer):
def __init__(self, node):
GenServer.__init__(self,
node_name=node.node_name_,
accepted_calls=["handle_call"])
node.register_name(self, Atom("py_process"))
def handle_call(self, args):
print(type(args))
print(args)
return "foo"
def main():
eng = GeventEngine()
node = Node(node_name="[email protected]", cookie="COOKIE", engine=eng)
PyGenServer(node)
eng.run_forever()
if __name__ == "__main__":
main()
and start it:
$ PYTHONPATH=./Pyrlang PYRLANG_ENABLE_LOG_FORMAT=1 PYRLANG_LOG_LEVEL=DEBUG python3 py_app/first.py
I'll have an iex shell running, using same cookie, and started like:
iex --name [email protected] --cookie COOKIE
I'll have receiver defined in my Elixir side like:
receiver = {:py_process, :"[email protected]"}
Problematic entries
1 - Returned value
iex([email protected])15> r = GenServer.call(receiver, {:handle_call, 5})
'foo'
iex([email protected])17> i r
Term
'foo'
Data type
List
Description
This is a list of integers that is printed as a sequence of characters
delimited by single quotes because all the integers in it represent valid
ASCII characters. Conventionally, such lists of integers are referred to
as "charlists" (more precisely, a charlist is a list of Unicode codepoints,
and ASCII is a subset of Unicode).
Raw representation
[102, 111, 111]
Reference modules
List
Implemented protocols
IEx.Info, Inspect, String.Chars, List.Chars, Collectable, Enumerable
Even though, in Python code I'm returning string "foo", this is visible to Elixir as list of code points, while string/binary was expected.
iex([email protected])24> is_binary(r)
false
2 - Recognising sent data
iex([email protected])18> GenServer.call(receiver, {:handle_call, [1, 2, 3]})
I was struggling to implement handler for the data, then discovered, what Python sees is not exactly what I was expecting it to see, from the logs:
2018-11-07 08:29:30,552 [pyrlang] node:247: Send local reg=py_process receiver=<__main__.PyGenServer object at 0x10d343128> msg=(atom'$gen_call', (<3.111.0 @ [email protected]>,Ref<3,202040,3575644163,1815979671>@[email protected]), (atom'handle_call', '\x01\x02\x03'))
2018-11-07 08:29:30,553 [pyrlang.OTP] gen_server:84: In call GenIncomingMessage((atom'handle_call', '\x01\x02\x03'))
2018-11-07 08:29:30,553 [pyrlang.OTP] gen_server:106: method=<bound method PyGenServer.handle_call of <__main__.PyGenServer object at 0x10d343128>>
<class 'str'> // Please note, it recognised a list of integers sent from Elixir as a stringm and...
// ... Python was unable to print the value of the argument, and this line is empty
2018-11-07 08:29:30,553 [pyrlang.OTP] gen_server:108: Replying with result=foo
as contrary, when I send a tuple with the same data, like:
iex([email protected])18> GenServer.call(receiver, {:handle_call, {1, 2, 3}})
on Python side, it looks correct:
2018-11-07 08:29:30,575 [pyrlang] node:494: demonitor orig=<3.111.0 @ [email protected]> target=<1.0.2 @ [email protected]> ref=Ref<3,202040,3575644163,1815979671>@[email protected]
2018-11-07 08:32:44,895 [pyrlang] node:247: Send local reg=py_process receiver <__main__.PyGenServer object at 0x10d343128> msg=(atom'$gen_call', (<3.111.0 @ [email protected]>,Ref<3,202050,3575644163,1815979671>@[email protected]), (atom'handle_call', (1, 2, 3)))
2018-11-07 08:32:44,895 [pyrlang.OTP] gen_server:84: In call GenIncomingMessage((atom'handle_call', (1, 2, 3)))
2018-11-07 08:32:44,895 [pyrlang.OTP] gen_server:106: method=<bound method PyGenServer.classify of <__main__.PyGenServer object at 0x10d343128>>
<class 'tuple'> // this time it recognises properly, and...
(1, 2, 3) // ... it printed the arg correctly
2018-11-07 08:32:44,895 [pyrlang.OTP] gen_server:108: Replying with result=foo
3 - Decoding data, that crashes
In Elixir:
iex([email protected])22> GenServer.call(receiver, {:handle_call, [3.5]})
** (exit) exited in: GenServer.call({:py_process, :"[email protected]"}, {:handle_call, [3.5]}, 5000)
** (EXIT) no connection to [email protected]
(elixir) lib/gen_server.ex:924: GenServer.call/3
and in Python logs:
2018-11-07 09:19:58,349 [pyrlang] node:162: Node [email protected] connected
2018-11-07 09:19:58,382 [pyrlang] gevent_engine:176: Exception: Traceback (most recent call last):
File "/Users/.../pyex_project/Pyrlang/pyrlang/async/gevent_engine.py", line 173, in _serve_loop
_read_loop(proto=proto, sock=sock)
File "/Users/.../pyex_project/Pyrlang/pyrlang/async/gevent_engine.py", line 206, in _read_loop
collected1 = proto.on_incoming_data(collected)
File "/Users/.../pyex_project/Pyrlang/pyrlang/dist/base_dist_protocol.py", line 154, in on_incoming_data
if self.on_packet(packet):
File "/Users/.../pyex_project/Pyrlang/pyrlang/dist/in_dist_protocol.py", line 50, in on_packet
return self.on_packet_connected(data)
File "/Users/.../pyex_project/Pyrlang/pyrlang/dist/base_dist_protocol.py", line 366, in on_packet_connected
(msg_term, tail) = codec.binary_to_term(tail)
File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 113, in binary_to_term
return binary_to_term_2(data[1:], options)
File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 262, in binary_to_term_2
term1, tail = binary_to_term_2(tail)
File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 262, in binary_to_term_2
term1, tail = binary_to_term_2(tail)
File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 245, in binary_to_term_2
term1, tail = binary_to_term_2(tail)
File "/Users/.../pyex_project/pyex_env/lib/python3.6/site-packages/term/py_codec_impl.py", line 405, in binary_to_term_2
raise PyCodecError("Unknown tag %d" % data[0])
term.py_codec_impl.PyCodecError: Unknown tag 99
I know this is quite lenghty, but I was hoping to provide as much information as I could!
Let me know if there is anything else I could help with this issue!
Best regards, Paweł
Hi,
Sorry for the late response, I haven't managed to set notifications on this repo.
I've glanced through your issue, update me if I missed something.
I think that it's that a list of integers with values under 255 will be represented as a string. I'm not sure, but in erlang strings ar just list of integers:
2> [99,100].
"cd"
3>
could you try with a list of integers with higher values then 255 to see if I'm correct.
I see - thank you for responding @s2hc-johan. I'll give it a try within the next two days and get back to you 👍