Encode for foreign types?

[luxalpa]

Serde has #[serde(deserialize_with = "path")] in order to deserialize using a custom function. I'm not sure how easy that would be to implement in bitcode.

The issue I currently have is that I'm using IndexVec which I can't add derives to, and I'd like it to serialize (and deserialize) like a Vec.

[finnbear]

bitcode currently doesn't support 3rd party encoding implementations because the encoding machinery is internal and unstable. Without writing your own encode implementation, there would be nothing to point #[bitcode(encode_with = "???")] at.

For now, see the serde feature flag. The only disadvantage is slightly less efficiency (both in size - because no more hints and less efficient enums - and speed):

• #[bitcode(with_serde)] is a good option for individual fields of foreign types within a bitcode::Encode struct.
• An alternative is to only use bitcode::serialize instead of bitcode::encode.

We are also debating whether to have bitcode feature flags for crates like indexmap/index_vec/glam/etc. and/or a way to encode T: IntoIter<Item = I> + FromIterator<I>.

[luxalpa]

It would be good to have a way to wrap other data structures. Currently trying to encode leptos' RwSignal structs which need calls to value and new - it would be nice if I could just 'forward' the encode/decode to the inner value (in this case the return value of the function call).

[rjobanp]

For anyone else who comes across this issue -- #[bitcode(with_serde)] is no longer available in 0.6.0 so the only alternative for encoding foreign types is to use the serde feature with bitcode::serialize. As far as I can tell the serde methods are a bit slower than bitcode::Encode and don't expose a way to reuse allocations across encode calls.

[luxalpa]

I was actually able to just manually implement and defer the encode/decode functions on the respective foreign types and for now the better workaround for handling foreign types like chrono or indexvec is to just fork those packages. Still, I think it would be very nice to have a way to specify an external encoder. Maybe I'll try to add that functionality when I find the time for it. So far, these workarounds are sufficient for me.

Haven't tried to update to 0.6 yet though.

[juh9870]

bitcode currently doesn't support 3rd party encoding implementations because the encoding machinery is internal and unstable. Without writing your own encode implementation, there would be nothing to point #[bitcode(encode_with = "???")] at.

One way to solve this would be to allow deferring encoding/decoding to a different type.

This way, all the encoding/decoding logic remains an implementation detail of the bitcode, and all the user can do is to "replace" a foreign type with a custom type at encoding/decoding time.

Via From trait

#[derive(Debug, Encode, Decode)]
pub struct MainType {
    a: usize,
    #[bitcode(with=ForeignTypeMock)]
    b: ForeignType,
}

#[derive(Debug, Encode, Decode)]
struct ForeignTypeMock {...}

impl From<ForeignType> for ForeignTypeMock {...}
impl From<ForeignTypeMock> for ForeignType {...}

serde-like format:

#[derive(Debug, Encode, Decode)]
pub struct MainType {
    a: usize,
    #[bitcode(with=foreign_mock)]
    b: ForeignType,
}

// SomeSupportedType implements Encode and Decode
mod foreign_mock {
    pub fn encode(data: ForeignType) -> SomeSupportedType {...}
    pub fn decode(data: SomeSupportedType) -> ForeignType {...}
}

[sargarass]

I ended up making something like this to bypass the restriction. It does use a private API and even probably works, but it could break at any time in the future. Given that the format is unstable and we're tied to a specific version, this seems like the viable option.

I wish there were an alternative with a #[bitcode(encode_with="...")] and #[bitcode(decode_with="...")] like solution.

impl_encodable! and BitcodeEncodable

use std::{fmt::Debug, ops::Deref};

use derive_more::{Deref, From};

#[derive(Deref, From, Copy, Clone, Debug, PartialOrd, PartialEq)]
#[repr(transparent)]
pub struct BitcodeEncodable<T>(T);

macro_rules! impl_encodable {
    (encoder: $wrapping_type:ty, $with:ty, $encode: expr) => {
        const _: () = {
            impl ::bitcode::Encode for BitcodeEncodable<$wrapping_type> {
                type Encoder = Encoder;
            }

            #[derive(Default)]
            pub struct Encoder(<$with as ::bitcode::Encode>::Encoder);

            impl ::bitcode::__private::Buffer for Encoder {
                fn collect_into(&mut self, out: &mut Vec<u8>) {
                    self.0.collect_into(out)
                }

                fn reserve(&mut self, additional: ::std::num::NonZeroUsize) {
                    self.0.reserve(additional)
                }
            }

            impl ::bitcode::__private::Encoder<BitcodeEncodable<$wrapping_type>> for Encoder {
                fn encode(&mut self, t: &BitcodeEncodable<$wrapping_type>) {
                    self.0.encode(&$encode(&t.0))
                }
            }
        };
    };
    (decoder: $wrapping_type:ty, $lt:lifetime, $with:ty, $decode: expr) => {
        impl_encodable! { _gen_decoder_impl: $wrapping_type, $lt, &$lt $with, $decode }
    };
    (decoder: $wrapping_type:ty, $with:ty, $decode: expr) => {
        impl_encodable!{ _gen_decoder_impl: $wrapping_type, 'de, $with, $decode }
    };
    (_gen_decoder_impl: $wrapping_type:ty, $lt:lifetime, $with:ty, $decode: expr) => {
        const _: () = {
            impl<$lt> ::bitcode::Decode<$lt> for BitcodeEncodable<$wrapping_type> {
                type Decoder = Decoder<$lt>;
            }

            #[derive(Default)]
            pub struct Decoder<$lt>(<$with as ::bitcode::Decode<$lt>>::Decoder);

            impl<$lt> ::bitcode::__private::View<$lt> for Decoder<$lt> {
                fn populate(&mut self, input: &mut &$lt [u8], length: usize) -> bitcode::__private::Result<()> {
                    self.0.populate(input, length)
                }
            }

            impl<$lt> ::bitcode::__private::Decoder<$lt, BitcodeEncodable<$wrapping_type>> for Decoder<$lt> {
                fn decode_in_place(&mut self, out: &mut ::std::mem::MaybeUninit<BitcodeEncodable<$wrapping_type>>) {
                    out.write(BitcodeEncodable($decode(self.0.decode())));
                }
            }
        };
    };
}

usage example

use bitcode::{Decode, Encode};

use wrappers::BitcodeEncodable;
use foreigner::{CompactString, FixedPoint, Timestamp};

impl_encodable!(encoder: CompactString, str, CompactString::as_str);
impl_encodable!(decoder: CompactString, &'de str, |s: &str| CompactString::from(s));

impl_encodable!(encoder: Timestamp, u64, |val: &Timestamp| val.as_nanos());
impl_encodable!(decoder: Timestamp, u64, |nanos: u64| Timestamp::from_nanos(nanos));

impl_encodable!(encoder: FixedPoint, i128, |val: &FixedPoint| *val.as_bits());
impl_encodable!(decoder: FixedPoint, i128, |val: i128| FixedPoint::from_bits(val));

#[test]
fn wrapper_works() {
    #[derive(Encode, Decode, PartialOrd, PartialEq, Debug)]
    struct T {
        s: u32,
        a: BitcodeEncodable<CompactString>,
        c: BitcodeEncodable<Timestamp>,
        d: BitcodeEncodable<FixedPoint>,
        e: u32,
    }

    let d: FixedPoint = FixedPoint::from_bits(321);
    let test = T {
        s: 0xC0DEDEAD,
        a: CompactString::from("test").into(),
        c: Timestamp::from_nanos(123).into(),
        d: d.into(),
        e: 0xDEADC0DE,
    };

    let bits = bitcode::encode(&test);
    let decoded: T = bitcode::decode(&bits).expect("decoded");
    assert_eq!(test, decoded);
}

// mock foreigner types for a codebase
mod foreigner {
    #[derive(Debug, Clone, PartialOrd, PartialEq)]
    pub struct CompactString(String);

    impl CompactString {
        pub fn as_str(&self) -> &str {
            &self.0
        }

        pub fn from(str: &str) -> Self {
            Self(String::from(str))
        }
    }

    #[derive(Debug, Copy, Clone, PartialOrd, PartialEq)]
    pub struct Timestamp(u64);

    impl Timestamp {
        pub fn as_nanos(self) -> u64 {
            self.0
        }

        pub fn from_nanos(val: u64) -> Self {
            Self(val)
        }
    }

    #[derive(Debug, Copy, Clone, PartialOrd, PartialEq)]
    #[repr(transparent)]
    pub struct FixedPoint(i128);

    impl FixedPoint {
        pub fn as_bits(&self) -> &i128 {
            &self.0
        }

        pub fn from_bits(val: i128) -> Self {
            Self(val)
        }
    }
}