Which serializer to use when using with both Commanded and jsonb?

[datafoo]

A note was recently added in the Getting started guide:

Note: To use an EventStore with Commanded you should configure the event store to use Commanded's JSON serializer which provides additional support for JSON decoding:

config :my_app, MyApp.EventStore, serializer: Commanded.Serialization.JsonSerializer

The same guide also states:

Using the jsonb data type

[…] To enable native JSON support you need to configure your event store to use the jsonb data type. You must also use the EventStore.JsonbSerializer serializer, to ensure event data and metadata is correctly serialized to JSON, […]

So, what is the recommended configuration when using the event store with both Commanded and jsonb? Is the following correct?

config :my_app, MyApp.EventStore,
  column_data_type: "jsonb",
  serializer: Commanded.Serialization.JsonSerializer,
  types: EventStore.PostgresTypes

[slashdotdash]

The Commanded.Serialization.JsonSerializer serializer will not work with the jsonb data type.

Instead, you can use this JsonbSerializer module which supports jsonb data type and the JSON decoding protocol. You will need to copy the module into your own app and reference it.

[datafoo]

I was using the JsonbSerializer initially. But in the process of migrating to Commanded 1.0.0, I tried to use Commanded.Serialization.JsonSerializer instead and everything just worked with the following configuration.

config :my_app, MyApp.EventStore,
  column_data_type: "jsonb",
  serializer: Commanded.Serialization.JsonSerializer,
  types: EventStore.PostgresTypes

It also works with JsonbSerializer.

So, how is it not supposed to work?

[slashdotdash]

The migration from Poison to Jason for JSON serialization might explain why it's working. I will need to investigate. Maybe we can now deprecate the JsonbSerializer.

[ItsRaWithTheH]

But in the process of migrating to Commanded 1.0.0, I tried to use Commanded.Serialization.JsonSerializer instead and everything just worked with the following configuration.

I would say it actually works better with the Commanded.Serialization.JsonSerializer rather than the Eventstore.JsonbSerializer.

When I use the jsonb serializer. events with maps that have nested string keys, breaks on deserialization as it recursively runs String.to_existing_atom(key) and the nested string keys may not be existing atoms (rightly so).

[jsmestad]

@slashdotdash I am hitting the error that @ItsRaWithTheH mentioned:

When I use the jsonb serializer. events with maps that have nested string keys, breaks on deserialization as it recursively runs String.to_existing_atom(key) and the nested string keys may not be existing atoms (rightly so).

How can I make this work? The nested string keys, is because the event contains a "raw" field which is passed through by the caller. However, I cannot get deserialize to work.

It is also worth mentioning that this bug is only seen in tests when switching to a PG backend, it does not get reproduced when using the InMemory adapter. 😕

Here is an example, using commanded_messaging in this case:

defmodule Commands.MyCommand do
  use Commanded.Command,
    item_ref: :string,
    raw: :map
end

defmodule Events.MyEvent do
  use Commanded.Event,
    from: Commands.MyCommand
end

# Commands.MyCommand.new(%{item_ref: 1, data: %{"foo" => "bar"}}) # OK
# Commands.MyCommand.new(%{item_ref: 1, data: %{"foo" => %{"baz" => "bar"}}}) # FAIL

[philiph]

Coming in kind of late here!

@datafoo you said:

I was using the JsonbSerializer initially. But in the process of migrating to Commanded 1.0.0, I tried to use Commanded.Serialization.JsonSerializer instead and everything just worked with the following configuration.

I'm in the process of upgrading an app from commanded 0.19.1 to 1.1.1, and as part of this also upgrading from eventstore 0.17.0 to 1.1.0. The app was already using the jsonb column type for the data and metadata columns on the events table in Postgres, and I wanted to keep it this way

I found that if I specified the Commanded.Serialization.JsonSerializer when the existing event store Postgres DB already had jsonb column types (for the data and metadata columns on the events table), it appeared to work fine at first glance when emitting events. However upon inspection of the database the event data was getting converted to a string (containing JSON) and saved into the data JSONB column, instead of the event's data actually getting saved as JSONB. From what I remember, deserialization from the DB did not work correctly. You may want to check this isn't happening (if it is, you'd be much better off just using a json column type).

So, I did end up using the JsonbSerializer mentioned by @slashdotdash above:

Instead, you can use this JsonbSerializer module which supports jsonb data type and the JSON decoding protocol. You will need to copy the module into your own app and reference it.

Hopefully this helps someone.