Serializing integral types - built-in aliases for `std::size_t`

[omartijn]

I'm having an issue with serialization of std::size_t with libc++ (the standard library coming with clang).

It seems that on most platforms, this is caught by the std::uint64_t variant, but when compiling with libc++ (the default on macOS) things go awry, because std::size_t is aliased to unsigned long, but std::uint64_t is not.

I could easily add an overload for unsigned long of course:

RW_JSON_VALUES( unsigned long, IsUint64, GetUint64, SetUint64 )

This fixes the compilation issues when using libc++, but of course it breaks elsewhere, because it's considered a redefinition of the same function (because the type aliases to std::uint64_t, which already has an overload).

I can't think of a nice solution to this, I hope you'll have more luck!

[eao197]

Hi!

It seems that the root of the problem is the use of platform-specific type std::size_t instead of std::uint64_t or something like that. But I think that you can't simply change std::size_t in data structures by something else.

So if we're speaking about handling that case at json_dto level then the simplest way could be the use of custom Reader_Writer for fields of type std::size_t. Something like that:

struct size_t_serializer
{
	void read( std::size_t & v, const rapidjson::Value & from ) const
	{
		using json_dto::read_json_value;

		std::uint64_t actual;
		read_json_value( actual, from );

		v = static_cast<std::size_t>(actual);
	}

	void
	write(
		const std::size_t & v,
		rapidjson::Value & to,
		rapidjson::MemoryPoolAllocator<> & allocator ) const
	{
		using json_dto::write_json_value;

		const std::uint64_t actual{v};
		write_json_value( actual, to, allocator );
	}
};
...
struct some_your_data_struct
{
	std::size_t payload_size_;
	std::string payload_;

	template< typename Json_Io >
	void json_io( Json_Io & io )
	{
		io & json_dto::mandatory( size_t_serializer{}, "size", payload_size_ )
			& json_dto::mandatory( "payload", payload_ );
	}
};

Another possible solution is new functionality added in v.0.2.12 (customization points in the form of binder_data_holder_t, binder_read_from_implementation_t and binder_write_to_implementation_t). It is similar to custom Reader_Writer, but requires more code.

[omartijn]

Thanks, @eao197

It's a shame that there's no "nice" way to do it, but it looks like that's due to the weird typedefs provided by the standard. To me, it'd make more sense if the standard declared that std::size_t should alias to std::uint116_t, std::uint32_t or std::uint64_t for 16, 32 and 64-bit architectures.

Would it be a solution to make the functions for numeric conversions templated, and then constrain them with SFINAE on being integral and having the correct size? Something like this:

template< typename type >
std::enable_if_t< std::is_integral_v< type > && std::is_unsigned_v< type > && sizeof( number_type ) == 4 >
read_json_value( type& value, const rapidjson::Value& object )
{
    if( object.IsUint32() )
        v = object.GetUint32();
    else
        throw ex_t{ "value is not Uint32" };
}

This would work for any unsigned type with 32-bits. If we define these templates for all relevant numeric types, we should catch them no matter whether they're aliased to unsigned long, unsigned long long, as long as they have the right size.

We can probably update the macros used for defining these functions to generate templated versions instead.

[eao197]

Would it be a solution to make the functions for numeric conversions templated, and then constrain them with SFINAE on being integral and having the correct size?

I don't think it has the sense if a user is able to use fixed-size types like std::uint32_t.

From my experience in serializing/deserializing data in various formats, the only robust method is to use fixed-size types for data members. It means std::uint32_t instead of unsigned int on 32- and 64-bit platforms, std::uint64_t instead of unsigned long and so on. That approach solves a lot of problems accessing serialized data on different platforms and from different languages.

Complex template-based implementations of read_json_value (and similar functions) will add a lot more hidden magic to json_dto implementation and I think it's not a good thing.

My opinion is that the usage of fixed-size types like std::unit32_t and own strong typedefs for types like std::size_t (which can be different of different platforms) is much better and more obvious and straightforward.

[omartijn]

Well, the issue with std::size_t is that it is commonly used for for things like container sizes. Sometimes you want to only serialize something like that directly. It's convenient if that "just works".

This goes especially if you're using non-intrusive json_io. I find myself fighting with json_dto because on macOS it chokes on the size_t being used, so my simple, declarative mapping doesn't work and I have to explicitly start casting types.

[eao197]

Well, the issue with std::size_t is that it is commonly used for for things like container sizes.

It's normal for in-process usage but can potentially lead to surprises when it's used in serialization/deserialization.

But I understand your case. Unfortunately, I don't see a good solution now. Let me return to the issue when I'll have some more time to think.

[eao197]

@omartijn Can you provide a small example of your code you have problems with? Such an example will allow me to understand better what you have and what you want to achieve.

[eao197]

It seems that there are several ways to cope with that problem. All of them require to #include not json_dto/pub.hpp directly, but your helper header file. Let say that file has the name json_dto_wrapper.hpp. So you have to write:

#include <json_dto_wrapper.hpp> // or "json_dto_wrapper.hpp

instead of

#include <json_dto/pub.hpp>

In the simplest case json_dto_wrapper.hpp can looks like:

#pragma once

#include <json_dto/pub.hpp>

#if <check for Apple platform here>

namespace json_dto
{

// Introduce overloads for unsigned long.
RW_JSON_VALUES( unsigned long, IsUint64, GetUint64, SetUint64 )

}

#endif  

A more complex way is to introduce some SFINAE-protected function templates into json_dto namespace. Those templates have to be chosen by the compiler if it can't find appropriate non-templated functions. In that case json_dto_wrapper.hpp can looks like:

#pragma once

#include <json_dto/pub.hpp>

namespace json_dto
{

template< typename type >
std::enable_if_t< std::is_integral_v< type > && std::is_unsigned_v< type > && sizeof( type ) == sizeof( std::uint32_t ) >
read_json_value( type& value, const rapidjson::Value& object )
{
    if( object.IsUint32() )
        v = object.GetUint32();
    else
        throw ex_t{ "value is not Uint32" };
}
... // Similar version of write_json_value 
}

NOTE. I don't check this approach, but hope it should work.

And in that case, it is your duty to make sure that your type has the same length on 32- and 64-bit platforms.

The last way is to make your specialization of binder_read_from_implementation and binder_write_to_implementation customization points. An example of how it can be done can be found here. It's an overcomplicated proof-of-concept only and I don't advise using it in the production.

PS. We discussed that issue in our team and decided to not modify the source of json_dto. Becase overloads like shown here can lead to code that behaves differently on different platforms.

[omartijn]

I'm not sure how, but it seems I missed this response. I'm indeed using a check for libc++ now and adding the overload in that case, which isn't pretty, but I do understand your concerns.