Better ExtraProperties mapping for EF Core

[hikalkan]

We are currently using a custom value converter to map a string db field to ExtraPropertyDictionary object when the entity implements IHasExtraProperties interface.

While this is a good way to extend an entity on runtime, it still has come limitations. One of the important limitation is that we can not sort or filter based on the extra properties using EF Core API.

Beginning from EF Core 8, it supports to map owned properties to JSON columns with automatic Dictionary<string, object?> conversion support.

Example code snippets:

Entity:

public class Book : BasicAggregateRoot<Guid>, IHasExtraProperties
{
    public string Name { get; set; }

    public BookType Type { get; set; }

    public DateTime PublishDate { get; set; }

    public float Price { get; set; }
    
    public ExtraPropertyDictionary ExtraProperties { get; set; }

    public Book()
    {
        ExtraProperties = new ExtraPropertyDictionary();
    }
}

Map ExtraProperties to JSON as owned property (removing the configuration added by ABP framework):

builder.Entity<Book>(b =>
{
    b.ToTable(AbpSlnExtraPropDemoConsts.DbTablePrefix + "Books",
        AbpSlnExtraPropDemoConsts.DbSchema);
    b.ConfigureByConvention(); //auto configure for the base class props
    b.Property(x => x.Name).IsRequired().HasMaxLength(128);

    b.Ignore(x => x.ExtraProperties); // I am removing the configuration added by ABP framework
    b.OwnsOne(x => x.ExtraProperties, navigationBuilder =>
    {
        navigationBuilder.ToJson();
        navigationBuilder.Property<bool>("IsAvailable");
        navigationBuilder.Property<int?>("Stock");
    });
});

Order by stock count:

var queryable = await _repository.GetQueryableAsync();
var query = queryable
    .OrderBy(x => EF.Property<bool>(x.ExtraProperties, "Stock"))
    .Skip(input.SkipCount)
    .Take(input.MaxResultCount);

Create a new entity

public async Task<BookDto> CreateAsync(CreateUpdateBookDto input)
{
    var book = ObjectMapper.Map<CreateUpdateBookDto, Book>(input);
    
    book.SetProperty("Stock", 42);
    book.SetProperty("NonExistingProp", "TEST"); // PROBLEM: CAN NOT SAVE!
    
    await _repository.InsertAsync(book);
    return ObjectMapper.Map<Book, BookDto>(book);
}

The main problem here is we can not get/set arbitrary properties. We should have to define them first in the OwnsOne configuration. Of course ABP will make it automatic when we define extra properties to entities, but if we didn't define it, we can not arbitrary set/get values, which is an important breaking change. If we can somehow solve this problem, this will be a better solution than the current one, since it allows us to filter/sort based on extra properties (and use them in linq statements) with db-level query optimization. If we can intercept the EF Core logic that serializes/deserializes the JSON, we may achieve it.

[maliming]

1. EF Core will use the ModificationCommand class to get the value of the property we configured (navigationBuilder.Property) and build a JSON object to write to the database.

   If we want to add an unconfigured property, we need to modify ModificationCommand class. But it's hard to intercept this class. Most of the classes/methods in it are sealed/private

https://github.com/dotnet/efcore/blob/release/9.0/src/EFCore.Relational/Update/ModificationCommand.cs#L886-L938

2. EF Core will throw an exception when ordering using the unconfigured properties. Only the properties that have been configured can be used.

System.InvalidOperationException: Translation of 'EF.Property<bool>(StructuralTypeShaperExpression(
    StructuralType: Volo.Abp.Data.ExtraPropertyDictionary
    ValueBufferExpression: b.ExtraProperties Q-> 
    IsNullable: False), "NonExistingProp")' failed. 

Either the query source is not an entity type, or the specified property does not exist on the entity type.

---

In summary, we must configure the properties that we might read or write in advance.

[hikalkan]

This would be a big breaking change and may result with data loss for some of the applications (actually, it doesn't delete the non-mapped data from the database table, but in some cases some kind of data loss may happen).

I am moving this to backlog until it is possible to implement this without breaking change.