match_dtype_to_c_struct does not work if struct has arrays as fields

[zehanort]

E.g. say that a kernel has this struct as an argument, with 2 arrays:

typedef struct {
    uint x[2];
    float y;
    uint z[3];
} my_struct;

I can not find a way to turn this definition into a proper type for PyOpenCL to work with it. My attempt:

>>> type = np.dtype([('x', cltypes.uint, (2,)), ('y', cltypes.float), ('z', cltypes.uint, (3,))])
>>> type
dtype([('x', '<u4', (2,)), ('y', '<f4'), ('z', '<u4', (3,))])
>>> my_type, my_struct_c_decl = cl.tools.match_dtype_to_c_struct(device, 't', my_type)
Traceback (most recent call last):
  File ".../lib/python3.7/site-packages/pytools/__init__.py", line 536, in _deco
    return func._memoize_dic[args]  # pylint: disable=protected-access
AttributeError: 'function' object has no attribute '_memoize_dic'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File ".../lib/python3.7/site-packages/pyopencl/compyte/dtypes.py", line 105, in dtype_to_ctype
    return self.dtype_to_name[dtype]
KeyError: dtype(('<u4', (2,)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<.../lib/python3.7/site-packages/decorator.py:decorator-gen-3>", line 2, in match_dtype_to_c_struct
  File ".../lib/python3.7/site-packages/pytools/__init__.py", line 539, in _deco
    result = func(*args)
  File ".../lib/python3.7/site-packages/pyopencl/tools.py", line 550, in match_dtype_to_c_struct
    c_fields.append("  %s %s;" % (dtype_to_ctype(field_dtype), field_name))
  File ".../lib/python3.7/site-packages/pyopencl/compyte/dtypes.py", line 107, in dtype_to_ctype
    raise ValueError("unable to map dtype '%s'" % dtype)
ValueError: unable to map dtype '('<u4', (2,))'

It seems like I can not use match_dtype_to_c_struct with any array-like type:

>>> t = np.dtype([('x', '<i8', (10,))])
>>> t
dtype([('x', '<i8', (10,))])
>>> t_c, t_c_decl = pyopencl.tools.match_dtype_to_c_struct(device, 't', t)
Traceback (most recent call last):
...
  File "/home/sotiris/.local/lib/python3.7/site-packages/pyopencl/compyte/dtypes.py", line 107, in dtype_to_ctype
    raise ValueError("unable to map dtype '%s'" % dtype)
ValueError: unable to map dtype '('<i8', (10,))'

Is there any other way to create the type of the above struct? Or to initialize it somehow?

[zehanort]

I tried registering each array type in my struct with get_or_register_dtype and retried with a redefined version consisting of the registered types I got, and now there is a different traceback:

Traceback (most recent call last):
  File ".../lib/python3.7/site-packages/pytools/__init__.py", line 536, in _deco
    return func._memoize_dic[args]  # pylint: disable=protected-access
AttributeError: 'function' object has no attribute '_memoize_dic'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<.../lib/python3.7/site-packages/decorator.py:decorator-gen-3>", line 2, in match_dtype_to_c_struct
  File ".../lib/python3.7/site-packages/pytools/__init__.py", line 539, in _deco
    result = func(*args)
  File ".../lib/python3.7/site-packages/pyopencl/tools.py", line 559, in match_dtype_to_c_struct
    cdl.add_dtype(field_dtype)
  File ".../lib/python3.7/site-packages/pyopencl/tools.py", line 468, in add_dtype
    self.add_dtype(field_dtype)
  File ".../lib/python3.7/site-packages/pyopencl/tools.py", line 466, in add_dtype
    for name, field_data in sorted(six.iteritems(dtype.fields)):
  File ".../lib/python3.7/site-packages/six.py", line 589, in iteritems
    return iter(d.items(**kw))
AttributeError: 'NoneType' object has no attribute 'items'

[zehanort]

And something more, I can not make it work with a plain array either, because arrays do not have the attribute fields:

>>> t = np.dtype((np.int, 10))
>>> t
dtype(('<i8', (10,)))
>>> x = pyopencl.tools.match_dtype_to_c_struct(device, 'arr', t)
Traceback (most recent call last):
  File ".../lib/python3.7/site-packages/pytools/__init__.py", line 536, in _deco
    return func._memoize_dic[args]  # pylint: disable=protected-access
AttributeError: 'function' object has no attribute '_memoize_dic'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python3.7/site-packages/decorator.py:decorator-gen-3>", line 2, in match_dtype_to_c_struct
  File ".../lib/python3.7/site-packages/pytools/__init__.py", line 539, in _deco
    result = func(*args)
  File ".../lib/python3.7/site-packages/pyopencl/tools.py", line 544, in match_dtype_to_c_struct
    fields = sorted(six.iteritems(dtype.fields),
  File ".../lib/python3.7/site-packages/six.py", line 589, in iteritems
    return iter(d.items(**kw))
AttributeError: 'NoneType' object has no attribute 'items'

[inducer]

I think I simply must have not realized that it was necessary to consider that case when I wrote the struct alignment code. I'd be grateful for a PR.

[zehanort]

Just opened a PR, looking forward to your review!