msgpack icon indicating copy to clipboard operation
msgpack copied to clipboard

How to extend the library to new types?

Open PaoloOpluswork opened this issue 2 years ago • 4 comments

I have structs with fields a type msgpack does not understand. Fortunately the type has a simple way to be built and stored as sequence of bytes, so I expected to be fairly straightforward to extend the library to support it.

However, when I try to use it I get the error "msgpack : invalid code c7 decoding struct". 0xc7 is of course the code of Ext8 that I used since we are speaking of 40-70 bytes per element.

I have the feeling I am missing something obvious...

Here is the relevant code, minimized.

type MyEncoder struct {}

type MyDecoder struct {}

func (*MyEncoder) Code() int8 {
  return -3
}

func (*MyEncoder) Type() reflect.Type {
  return reflect.TypeOf(MyThingy)
}

func (*MyEncoder) CalcByteSize(value reflect.Value) (int, error) {
  m := value.Interface().(MyThingy)
  return 2 + len(m.ToBytes()), nil
}

func (c *MyEncoder) WriteToBytes(value reflect.Value, offset int, bytes *[]byte) int {
  m := []byte(value.Interface().(MyThingy).ToBytes())

  (*bytes)[offset] = msgpackDef.Ext8
  offset++

  (*bytes)[offset] = byte(c.Code())
  offset++

  (*bytes)[offset] = byte(len(m))
  offset++

  for _, c := range m {
  	(*bytes)[offset] = c
  	offset++
  }

  return offset
}

func (*MyDecoder) Code() int8 {
  return -3
}

func (c *MyDecoder) IsType(offset int, d *[]byte) bool {
  return (*d)[offset] == msgpackDef.Ext8 && (*d)[offset+1] == byte(c.Code())
}

func (*MyDecoder) AsValue(offset int, k reflect.Kind, d *[]byte) (interface{}, int, error) {
  nrBytes := int((*d)[offset+2])
  m := MyThingy{}
  mm, err := m.FromBytes((*d)[3 : 3+nrBytes])
  return mm, 3 + nrBytes, err
}

PaoloOpluswork avatar Jan 24 '24 05:01 PaoloOpluswork

I'm sorry for delay!

Have you already solved this issue? If not so, do you call msgpack.AddExtCoder in your code?

shamaton avatar Mar 11 '24 23:03 shamaton

I did not solve, and it was time ago and I do not remember exactly if I called that function, albeit since I did not mention in the report I think not. But I'll try again and let you know.

PaoloOpluswork avatar Mar 12 '24 00:03 PaoloOpluswork

Indeed I forgot to call AddExtCoder, once added it almost worked outright.

I forgot to use properly the offset in the AsValue function (not used in the m.FromBytes call and in return statement)

Before we close this issue, I have another question: should CalcByteSize count the code length? In other words, should it return 2 + len(m.ToBytes()) or 3 + len(m.ToBytes())?

PaoloOpluswork avatar Mar 12 '24 03:03 PaoloOpluswork

Yes. You should return the total byte size that the struct is encoded to msgpack format. see also: https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family

shamaton avatar Mar 15 '24 14:03 shamaton