bytes icon indicating copy to clipboard operation
bytes copied to clipboard

Enabling zero copy deserialization (e.g. splitting a `BytesMut` based on on a &[u8] owned by that `BytesMut`).

Open hgomersall opened this issue 4 years ago • 0 comments

Some serde Deserializers can do zero-copy &[u8] -> &[u8] conversion. This is obviously quite useful.

Unfortunately, there seems currently not to be a way in which a deserializer can zero copy from/into a BytesMut (for obvious reasons - it might be possible with specialization, but I haven't thought to carefully about it...).

The problem then is that you lose the benefits of having your data encapsulated in a BytesMut object, like passing to a downstream async block for further manipulation.

It's possible to split a BytesMut based on an extracted zero-copied slice fairly safely as follows:

// buf is the source `BytesMut`
// buf_slice is a slice from it (returned from e.g. the deserializer)
let size = buf_slice.len();
// The unsafe is just because `offset_from` is unsafe (mostly fairly benignly)
let offset = unsafe {
    buf_slice.as_ptr().offset_from(buf.as_ptr()) as usize
};
let mut buf = buf.split_off(offset);
let _ = buf.split_off(size);

Link to rust playground.

It would be good if this was somehow better supported. It might not be possible easily, but I thought it worth raising for discussion since it seems to be a missing piece.

hgomersall avatar Feb 19 '21 14:02 hgomersall