Attribute macro to generate de/serialization functions for fields of big array type

[dtolnay]

Like all of the standard library's traits, Serde's traits are limited to fixed size arrays up to an arbitrary maximum size. Serde defines Serialize and Deserialize impls for arrays up to size 32.

The current workaround for larger arrays in serde_big_array is workable but not ideal:

big_array! {
    BigArray;
    42, 300,
}

#[derive(Serialize, Deserialize)]
struct S {
    #[serde(with = "BigArray")]
    arr_a: [u8; 300],
    #[serde(with = "BigArray")]
    arr_b: [u8; 42],
    arr_small: [u8; 8],
}

It would be nicer to have an attribute macro that makes big arrays work by finding all fields of array type and inserting the appropriate serde(serialize_with = "...", deserialize_with = "...") functions (also generated by the attribute macro).

#[make_big_arrays_work]
#[derive(Serialize, Deserialize)]
struct S {
    arr_a: [u8; 300],
    arr_b: [u8; 42],
    arr_small: [u8; 8],
}

// generated code

#[derive(Serialize, Deserialize)]
struct S {
    #[serde(
        serialize_with = "big_array_serialize_S_arr_a",
        deserialize_with = "big_array_deserialize_S_arr_a",
    )]
    arr_a: [u8; 300],
    #[serde(
        serialize_with = "big_array_serialize_S_arr_b",
        deserialize_with = "big_array_deserialize_S_arr_b",
    )]
    arr_b: [u8; 42],
    arr_small: [u8; 8],
}

fn big_array_serialize_S_arr_a<S>(
    array: &[u8; 300],
    serializer: S,
) -> core::result::Result<S::Ok, S::Error>
where
    S: serde::Serializer,
{
    /* ... */
}

fn big_array_deserialize_S_arr_a<'de, D>(
    deserializer: D,
) -> core::result::Result<[u8; 300], D::Error>
where
    D: serde::Deserializer<'de>,
{
    /* ... */
}

/* ... */

The serialize_with attribute should only be emitted if there is a Serialize derive on the data structure, and deserialize_with should only be emitted if there is a Deserialize derive.

Neither attribute should be emitted for a field with array type with literal size that we can see is 32 or smaller.

Attributes do need to be emitted for all arrays of const size not visible to the macro, for example arr_unknown: [u8; BUFFER_SIZE].

Optionally, also support type aliased arrays by specifying the array size in an attribute.

pub const BUFSIZE: usize = 1024;
pub type Buffer = [u8; BUFSIZE];

#[make_big_arrays_work]
#[derive(Serialize, Deserialize)]
struct S {
    #[big_array(BUFSIZE)]
    buffer: Buffer,
}

[est31]

@dtolnay that would indeed be a bit more convenient. Overall I think that I personally don't want to spend too much time on the issue, given that hopefully this will get fixed by the language.

[est31]

#[make_big_arrays_work]

Doesn't this require proc_macro_hygiene? So this is not implementable in the stable language, is it?

[dtolnay]

I don't think it would require proc_macro_hygiene -- why? Attribute macros on structs have been stable since 1.30.0.

[est31]

Oh I see. Thanks for clarifying that. It's a bit of a tricky situation with some things still unstable but some things being stabilized. One could work with that I guess.

[uint]

I figured this would be a great way to dig into proc macros, so I played around with it a bit. Who'd guess I'd have a PoC so soon: https://github.com/uint/serbia

For now, it only works on structs (tuple or regular). I guess I'm on it!

[est31]

serde_with has also recently gained big array support: https://github.com/jonasbb/serde_with/pull/272

Their MSRV is 1.51.

Support is a bit more comprehensive than serde-big-array. I need to investigate if there is a reason to keep serde-big-array around or whether I should deprecate it in favour of serde_with.

[uint]

So I've been looking at implementing this thing:

pub const BUFSIZE: usize = 1024;
pub type Buffer = [u8; BUFSIZE];

#[make_big_arrays_work]
#[derive(Serialize, Deserialize)]
struct S {
    #[big_array(BUFSIZE)]
    buffer: Buffer,
}

I can parse BUFSIZE as a syn::ExprPath or just a syn::Path, but I'm not sure there's a way to get the value of the underlying constant at macro expansion time. I've read somewhere that constants are evaluated after macro expansion. Is there some trick to this?

[uint]

@dtolnay I've been hacking at this. At this point, I think Serbia is pretty usable, but would love feedback.