asdf icon indicating copy to clipboard operation
asdf copied to clipboard

storing non `ndarray` types as blocks

Open CagtayFabry opened this issue 5 years ago • 2 comments

Is there a way to store any other types besides ndarray as a block in an ASDF file?

I am looking for a simple way to serialize (and tag) bytes, bytearray, BytesIO or similar without going through numpy

Could this be done in an extension library or would this need changes in asdf?

CagtayFabry avatar Dec 10 '20 14:12 CagtayFabry

Currently the code requires that anything written to a block be wrapped in ndarray, but I think it's possible to wrap bytes in ndarray without copying the memory:

In [21]: some_bytes = b'a' * 1024

In [22]: array = np.frombuffer(some_bytes, dtype=np.uint8)

In [23]: array.flags
Out[23]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : False
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

Now that array can be assigned to a block using the BlockManager.find_or_create_block_for_array method.

On the way back I don't know how to retrieve the bytes except to call ndarray.tobytes, which does make a copy.

eslavich avatar Apr 02 '21 20:04 eslavich

yes that is a possible workaround we used before

However I was looking for a possibility to explicitly differentiate from ndarray but in this case I will probably just wrap the numpy calls in a custom tag/object

CagtayFabry avatar Apr 27 '21 17:04 CagtayFabry