fivem icon indicating copy to clipboard operation
fivem copied to clipboard
verified publisher icon citizenfx

feat(clrcore-v2): Move to custom MsgPack serializer

Open thorium-cfx opened this issue 1 year ago • 1 comments

Goal of this PR

Enable (de)serialization to and from custom types on events, exports, nui callbacks, and other refFuncs within the mono rt2 runtime. A highly requested feature for the C# environments, as a lot of extra work and workarounds are currently applied to compensate for the lack there-of.

  1. Optional parameters are also supported with this update.
  2. Performance is roughly 30% faster for single listeners, making it slower for 3+ listeners for each event (why would you even do this?).

Notes:

  1. This is the last big update for the mono rt2 scripting runtime before we exit the pilot stage.
  2. Issues are expected; unit tests are put in place for some situations, but not all and/or unforeseen use-cases are tested yet.

How is this PR achieving the goal

Allows for in place conversion of serialized data.

  • Replaces deserializer on events, exports, nui callbacks, and other refFuncs with our custom and more extensive msgpack serialization.
  • Has support for:
    • Class/struct field/property (de)serialization.
    • Class/struct constructor deserialization.
    • Dictionary<K, V>, List<T>, and T[].
    • Primitives.
  • Add msgpack-cs submodule (production branch)
  • Temporary Player (de)serialization
  • Command remapper no longer allows reordering of parameters, only remote/player id

Example:

[EventHandler("myEvent")]
public static async Coroutine GimmeAll(int a)
	=> Debug.WriteLine($"GimmeAll1 {a}");

[EventHandler("myEvent")]
public static async Coroutine GimmeAll(string a, int b)
	=> Debug.WriteLine($"GimmeAll2 {a} {b}");

[EventHandler("myEvent")]
public static async Coroutine GimmeAll(string a, int b, string c = "Hey")
	=> Debug.WriteLine($"GimmeAll3 {a} {b} {c}");

[EventHandler("myEvent")]
public static async Coroutine GimmeAll(int a, string b, string c = "Oh", int d = 678)
	=> Debug.WriteLine($"GimmeAll4 {a} {b} {c} {d}");

[EventHandler("myEvent")]
public static async Coroutine GimmeAll(int a, Player b, string c = "Oh", int d = 678)
	=> Debug.WriteLine($"GimmeAll5 {a} {b} {c} {d}");

// Trigger it!
[Command("Gimme")]
public async Coroutine Gimme(uint source, object[] objects, string raw)
	=> Events.TriggerServerEvent("myEvent", 1234, "5678");

gives the following output:

[    script:csharp_v2] GimmeAll1 1234
[    script:csharp_v2] GimmeAll2 1234 5678
[    script:csharp_v2] GimmeAll3 1234 5678 Hey
[    script:csharp_v2] GimmeAll4 1234 5678 Oh 678
[    script:csharp_v2] GimmeAll5 1234 Player(5678) Oh 678

This PR applies to the following area(s)

ScRT: C#

Successfully tested on

FXServer and

Game builds: N/A

Platforms: Windows

Checklist

  • [x] Code compiles and has been tested successfully.
  • [x] Code explains itself well and/or is documented.
  • [x] My commit message explains what the changes do and what they are for.
  • [x] No extra compilation warnings are added by these changes.

Fixes issues

closes thorium-cfx/mono_v2_get_started#20

Thanks / Contributors

A big thanks goes out @manups4e, showing me how his FxEvents project used a completely different approach, this became the base and eventually the way forward on MsgPack (de)serialization for the mono v2 scripting environment.

thorium-cfx avatar May 15 '24 13:05 thorium-cfx

Ooo fancy!

Lorenc95 avatar May 16 '24 07:05 Lorenc95

Is it possible to provide an example of how to use the API.EventManagerPeekEvent as i am seeing this as still requiring an object to be passed as a reference ?

schuster19uk avatar Jun 01 '24 13:06 schuster19uk

@thorium-cfx Looks like this has broken the export system and every time an export is executed an error is thrown.

Example code:

[Export("test")]
public static string ExportE2()
{
    return "yeahh";
}

[    c-scripting-mono] Exception in Mono script environment: System.NotSupportedException: Requested serializer for MethodInfo.CustomAttributes of type System.Collections.Generic.IEnumerable`1[System.Reflection.CustomAttributeData] could not be found.
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializeMapBody (System.Type type, System.Reflection.Emit.ILGenerator g, CitizenFX.MsgPack.Detail.DynamicArray`1[T] members, System.Reflection.MethodInfo currentSerializer) [0x0023f] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:323
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializer (System.Type type, System.Reflection.Emit.TypeBuilder typeBuilder) [0x00117] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:170
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.Build (System.Type type) [0x000ce] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:79
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.CreateSerializer (System.Type type) [0x0008e] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:214
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.GetOrCreateSerializer (System.Type type) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:167
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializeMapBody (System.Type type, System.Reflection.Emit.ILGenerator g, CitizenFX.MsgPack.Detail.DynamicArray`1[T] members, System.Reflection.MethodInfo currentSerializer) [0x0015b] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:293
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializer (System.Type type, System.Reflection.Emit.TypeBuilder typeBuilder) [0x00117] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:170
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.Build (System.Type type) [0x000ce] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:79
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.CreateSerializer (System.Type type) [0x0008e] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:214
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.Serialize (CitizenFX.MsgPack.MsgPackSerializer serializer, System.Object obj) [0x0018a] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:143

[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.Serialize (System.Object v) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:70
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.Serialize (System.Object[] v) [0x0000d] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:256
[    c-scripting-mono]   at (wrapper dynamic-method) System.Object.MsgPackSerializer(CitizenFX.MsgPack.MsgPackSerializer,object)
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.Serialize (CitizenFX.MsgPack.MsgPackSerializer serializer, System.Object obj) [0x0017b] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:139

[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.Serialize (System.Object v) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:70
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.SerializeToByteArray (System.Object value) [0x00005] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:59
[    c-scripting-mono]   at CitizenFX.Core.Native.InPacket..ctor (System.Object obj) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Native\Types\NativeTypes.cs:80
[    c-scripting-mono]   at CitizenFX.Core.Native.InPacket.Serialize (System.Object obj) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Native\Types\NativeTypes.cs:81
[    c-scripting-mono]   at CitizenFX.Core.Native.InPacket.op_Implicit (System.Object[] obj) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Native\Types\NativeTypes.cs:82
[    c-scripting-mono]   at CitizenFX.Core._LocalFunction.Invoke (System.Object[] args) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Interop\LocalFunction.cs:46
[    c-scripting-mono]   at CitizenFX.Core.ExportsManager.IncomingRequest (System.String eventName, System.String sourceString, CitizenFX.Core.Binding origin, System.Byte* argsSerialized, System.Int32 serializedSize) [0x0003b] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Interop\ExportsManager.cs:34
[    c-scripting-mono]   at CitizenFX.Core.ScriptInterface.TriggerEvent (System.String eventName, System.Byte* argsSerialized, System.Int32 serializedSize, System.String sourceString, System.UInt64 hostTime, System.Boolean profiling) [0x0003c] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\ScriptInterface.cs:151
[    c-scripting-mono]   at (wrapper native-to-managed) CitizenFX.Core.ScriptInterface.TriggerEvent(string,byte*,int,string,ulong,bool,System.Exception&)
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializeMapBody (System.Type type, System.Reflection.Emit.ILGenerator g, CitizenFX.MsgPack.Detail.DynamicArray`1[T] members, System.Reflection.MethodInfo currentSerializer) [0x0023f] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:323
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializer (System.Type type, System.Reflection.Emit.TypeBuilder typeBuilder) [0x00117] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:170
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.Build (System.Type type) [0x000ce] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:79
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.CreateSerializer (System.Type type) [0x0008e] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:214
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.GetOrCreateSerializer (System.Type type) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:167
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializeMapBody (System.Type type, System.Reflection.Emit.ILGenerator g, CitizenFX.MsgPack.Detail.DynamicArray`1[T] members, System.Reflection.MethodInfo currentSerializer) [0x0015b] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:293
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.BuildSerializer (System.Type type, System.Reflection.Emit.TypeBuilder typeBuilder) [0x00117] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:170
[    c-scripting-mono]   at CitizenFX.MsgPack.Formatters.TypeFormatter.Build (System.Type type) [0x000ce] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\Formatters\TypeFormatter.cs:79
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.CreateSerializer (System.Type type) [0x0008e] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:214
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.Serialize (CitizenFX.MsgPack.MsgPackSerializer serializer, System.Object obj) [0x0018a] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:143

[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.Serialize (System.Object v) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:70
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.Serialize (System.Object[] v) [0x0000d] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:256
[    c-scripting-mono]   at (wrapper dynamic-method) System.Object.MsgPackSerializer(CitizenFX.MsgPack.MsgPackSerializer,object)
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackRegistry.Serialize (CitizenFX.MsgPack.MsgPackSerializer serializer, System.Object obj) [0x0017b] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackRegistry.cs:139

[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.Serialize (System.Object v) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:70
[    c-scripting-mono]   at CitizenFX.MsgPack.MsgPackSerializer.SerializeToByteArray (System.Object value) [0x00005] in C:\gl\builds\cfx-fivem\master\fxserver\vendor\msgpack-cs\MsgPack\MsgPackSerializer.cs:59
[    c-scripting-mono]   at CitizenFX.Core.Native.InPacket..ctor (System.Object obj) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Native\Types\NativeTypes.cs:80
[    c-scripting-mono]   at CitizenFX.Core.Native.InPacket.Serialize (System.Object obj) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Native\Types\NativeTypes.cs:81
[    c-scripting-mono]   at CitizenFX.Core.Native.InPacket.op_Implicit (System.Object[] obj) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Native\Types\NativeTypes.cs:82
[    c-scripting-mono]   at CitizenFX.Core._LocalFunction.Invoke (System.Object[] args) [0x00000] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Interop\LocalFunction.cs:46
[    c-scripting-mono]   at CitizenFX.Core.ExportsManager.IncomingRequest (System.String eventName, System.String sourceString, CitizenFX.Core.Binding origin, System.Byte* argsSerialized, System.Int32 serializedSize) [0x0003b] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\Interop\ExportsManager.cs:34
[    c-scripting-mono]   at CitizenFX.Core.ScriptInterface.TriggerEvent (System.String eventName, System.Byte* argsSerialized, System.Int32 serializedSize, System.String sourceString, System.UInt64 hostTime, System.Boolean profiling) [0x0003c] in C:\gl\builds\cfx-fivem\master\fxserver\code\client\clrcore-v2\ScriptInterface.cs:151
[    c-scripting-mono]   at (wrapper native-to-managed) CitizenFX.Core.ScriptInterface.TriggerEvent(string,byte*,int,string,ulong,bool,System.Exception&)Failed to execute event __cfx_export_monodb_testt - 80070057.

DaniGP17 avatar Jun 11 '24 18:06 DaniGP17

In case anyone decides to continue the work of this PR, I think that linking the crashes that were found when the changes of this PR were in effect could be useful: #2571 #2582 #2590 #2613 #2653

IstuntmanI avatar Dec 19 '24 22:12 IstuntmanI

If someone fixes all those bugs will someone from cfx decide to merge the changes?

DaniGP17 avatar Dec 20 '24 07:12 DaniGP17