UnicodeDecodeError using take_screen_shot_to_array()

[noxdafox]

This happens with both VirtualBox PUEL and VirtualBox OSE (setting sys.abiflags = '').

In [1]: import virtualbox
   ...: vbox = virtualbox.VirtualBox()
   ...: vm = vbox.find_machine(vbox.machines[0].name)
   ...: session = vm.create_session()
   ...: h, w, _, _, _, _ = session.console.display.get_screen_resolution(0)
   ...: png = session.console.display.take_screen_shot_to_array(0, h, w, virtualbox.library.BitmapFormat.png)
   ...: 
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
~/development/virtualenvs/pyvbox/lib/python3.5/site-packages/virtualbox/library_base.py in _call_method(self, method, in_p)
    194         try:
--> 195             ret = method(*in_params)
    196         except Exception as exc:

/usr/lib/virtualbox/sdk/bindings/xpcom/python/xpcom/client/__init__.py in takeScreenShotToArray(self, Param1, Param2, Param3, Param4)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
<ipython-input-1-a41323a6ea59> in <module>()
      4 session = vm.create_session()
      5 h, w, _, _, _, _ = session.console.display.get_screen_resolution(0)
----> 6 png = session.console.display.take_screen_shot_to_array(0, h, w, virtualbox.library.BitmapFormat.png)

~/development/virtualenvs/pyvbox/lib/python3.5/site-packages/virtualbox/library.py in take_screen_shot_to_array(self, screen_id, width, height, bitmap_format)
  23680             raise TypeError("bitmap_format can only be an instance of type BitmapFormat")
  23681         screen_data = self._call("takeScreenShotToArray",
> 23682                      in_p=[screen_id, width, height, bitmap_format])
  23683         return screen_data
  23684 

~/development/virtualenvs/pyvbox/lib/python3.5/site-packages/virtualbox/library_base.py in _call(self, name, in_p)
    184         method = self._search_attr(name)
    185         if inspect.isfunction(method) or inspect.ismethod(method):
--> 186             return self._call_method(method, in_p=in_p)
    187         else:
    188             return method

~/development/virtualenvs/pyvbox/lib/python3.5/site-packages/virtualbox/library_base.py in _call_method(self, method, in_p)
    210 
    211             if errobj.msg is None:
--> 212                 errobj.msg = getattr(exc, 'msg', getattr(exc, 'message'))
    213             raise errobj
    214         return ret

AttributeError: 'UnicodeDecodeError' object has no attribute 'message'

XPCOM client init file. xpcom_cli_init.txt

[sethmlarson]

Can you do me a favor and remove all the try-except logic within _call_method and post the actual traceback? I'm going to be rewriting this logic in v2.

[noxdafox]

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-a41323a6ea59> in <module>()
      4 session = vm.create_session()
      5 h, w, _, _, _, _ = session.console.display.get_screen_resolution(0)
----> 6 png = session.console.display.take_screen_shot_to_array(0, h, w, virtualbox.library.BitmapFormat.png)

~/development/virtualenvs/pyvbox/lib/python3.5/site-packages/virtualbox/library.py in take_screen_shot_to_array(self, screen_id, width, height, bitmap_format)
  23680             raise TypeError("bitmap_format can only be an instance of type BitmapFormat")
  23681         screen_data = self._call("takeScreenShotToArray",
> 23682                      in_p=[screen_id, width, height, bitmap_format])
  23683         return screen_data
  23684 

~/development/virtualenvs/pyvbox/lib/python3.5/site-packages/virtualbox/library_base.py in _call(self, name, in_p)
    184         method = self._search_attr(name)
    185         if inspect.isfunction(method) or inspect.ismethod(method):
--> 186             return self._call_method(method, in_p=in_p)
    187         else:
    188             return method

~/development/virtualenvs/pyvbox/lib/python3.5/site-packages/virtualbox/library_base.py in _call_method(self, method, in_p)
    193         in_params = [self._cast_to_valuetype(p) for p in in_p]
    194 
--> 195         ret = method(*in_params)
    196 
    197         return ret

/usr/lib/virtualbox/sdk/bindings/xpcom/python/xpcom/client/__init__.py in takeScreenShotToArray(self, Param1, Param2, Param3, Param4)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte

[sethmlarson]

It looks like for whatever reason XPCOM is converting the data into UTF-8 when it's probably just raw bytes or a bytearray. :/ It's a shame that traceback doesn't include any line numbers.

[noxdafox]

There is no line number as it's a single array of bytes. It contains the screenshot PNG image.

By PNG specifications, the first byte of the header must be 137 which in hexadecimal is 0x89.

The logic is trying to decode a byte array. It should actually return them as bytes and not str.

[sethmlarson]

I think this might be XPCOM's fault? Even though our logic incorrectly identifies the type octet as a str in the documentation we don't convert the output at all when we receive it.

[noxdafox]

From the trace is definitely hard to tell as we cannot see where it actually fails. Where can I see takeScreenShotToArray? Is it a Python function?

Kepp in mind that, in Python3, the distinction between bytes and str is critical.

[sethmlarson]

I'm aware of the distinction. :) I'm not sure what XPCOM does to load that function into existence because it's not defined within the Python bindings that come with the VirtualBox SDK.

[noxdafox]

Is the code available somewhere?

[sethmlarson]

It gets distributed with the VirtualBox SDK. XPCOM is the library that communicates with the VirtualBox process.

[noxdafox]

FYI: https://www.virtualbox.org/ticket/17364

[sethmlarson]

Thanks for debugging this! This means the fault may be outside our jurisdiction?

[noxdafox]

This is what happens with their reference example code.

In [1]: from vboxapi import VirtualBoxManager
In [2]: mgr = VirtualBoxManager(None, None)
In [3]: vbox = mgr.vbox
In [4]: vbox.findMachine('test')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-e0587a4315ea> in <module>()
----> 1 vbox.findMachine('test')

/usr/lib/virtualbox/sdk/bindings/xpcom/python/xpcom/client/__init__.py in findMachine(self, Param1)

TypeError: internal error in PyXPCOM, parameter must be a bytes object

[sethmlarson]

Oh sure, thanks for keeping this issue up to date. Let me know if you learn more!

[chroteus]

Just wanted to say that this bug is still present. If anyone needs screenshots, use VNC for now.

[guicho271828]

this bug is still present.

[guicho271828]

Regarding the method object that is throwing the utf-8 error, https://github.com/sethmlarson/virtualbox-python/blob/master/virtualbox/library_base.py#L201

It is a function that is built as a string inside xpcom/client/__init__.py (https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/client/init.py#L70) and gets compiled by compile in BuildMethod. https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/client/init.py#L130

I printed the string code (by inserting a print) which revealed that the function is

def takeScreenShotToArray(self, Param1, Param2, Param3, Param4):
    return XPTC_InvokeByIndex(self._comobj_, 28, (((128, 6, 0, 0, 0), (128, 6, 0, 0, 0), (128, 6, 0, 0, 0), (128, 6, 0, 0, 0), (64, 6, 0, 0, 0), (96, 148, 4, 4, 4)), (Param1, Param2, Param3, Param4)))

which matches the information from the stack trace:

> /usr/lib/virtualbox/sdk/bindings/xpcom/python/xpcom/client/__init__.py in takeScreenShotToArray(self, Param1, Param2, Param3, Param4)

XPTC_InvokeByIndex is from xpcom._xpcom package https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/client/init.py#L47

which is a compiled module in https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/module/_xpcom.cpp

You can find XPTC_InvokeByIndex here https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/module/_xpcom.cpp#L257

[guicho271828]

• MakePythonResult seems to be the crux. https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/module/_xpcom.cpp#L302
• It further calls MakeSinglePythonResult https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/VariantUtils.cpp#L2088
• which is located here https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/VariantUtils.cpp#L1850
• MakeSinglePythonResult uses type tag information (like in most dynamic type programming languages) of an object and dispatches based on the tag. Type tags are defined in nsXPTType https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/xpcom/reflect/xptinfo/public/xptinfo.h#L174 and the container is defined in nsXPTCVariant https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/xpcom/reflect/xptcall/public/xptcall.h
• UTF8 is dispatched in https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/VariantUtils.cpp#L1923
• Array is dispatched in https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/VariantUtils.cpp#L1990
• the implementation does not differentiate UTF8 and CSTRING. this might be the problem
• Array uses UnpackSingleArray . UnpackSingleArray has a dubious line https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/VariantUtils.cpp#L685 --- When python version is 3, it always converts to unicode

			  case nsXPTType::T_CHAR_STR: {
				char **pp = (char **)pthis;
				if (*pp==NULL) {
					Py_INCREF(Py_None);
					val = Py_None;
				} else
#if PY_MAJOR_VERSION <= 2
					val = PyString_FromString(*pp);
#else
					val = PyUnicode_FromString(*pp);     <<<<<<<
#endif
				break;
				}

[guicho271828]

maybe https://pypi.org/project/libvirt-python/ is a better choice

[pspcreateprocess]

According to https://www.virtualbox.org/ticket/19740 the exact location of this error is here: https://www.virtualbox.org/browser/vbox/trunk/src/libs/xpcom18a4/python/src/VariantUtils.cpp#L627

if (array_type == nsXPTType::T_U8)
#if PY_MAJOR_VERSION <= 2
                return PyString_FromStringAndSize( (char *)array_ptr, sequence_size );
#else
                return PyUnicode_FromStringAndSize( (char *)array_ptr, sequence_size ); // <----- 
#endif

With array of bytes in python3 PyBytes_FromStringAndSize must be called, instead of PyUnicode_FromStringAndSize.

[sethmlarson]

Yeah the issue is upstream, not with this library. Hopefully a fix will land and be released.

[guicho271828]

ORACLE...

[wrenchonline]

this bug is still present. Maybe like a libvirt using other languages to realize XPCOM is a better choice.