BulkMerge() inserts duplicated data

[PayteR]

I'm trying to make CSV importer with more than 1mil records. These records are often duplicated in the file and i try to use merge to ignore these duplicities. Problem is that it not works in this way and records are inserted in table multiple times, even when proper ColumnPrimaryKeyExpression is set. Check the example here https://dotnetfiddle.net/6H2A18.

What's the proper way to check duplicities on bulk merge for custom PrimaryKey? Thank you.

[JonathanMagnan]

Hello @PayteR ,

In this case, you can use the options AllowDuplicateKeys = true (Only the last record information is taken).

context.BulkMerge(customers, options => {
	options.ColumnPrimaryKeyExpression = c => c.Code;
	options.AllowDuplicateKeys = true;
});

https://dotnetfiddle.net/EzepgR

[PayteR]

Thank you @JonathanMagnan , it works on that fiddle, but when i tried that on my local installation, then it doesn't work. I have a similar testing table there, here are screenshots:

So i think it's a bug, either in EF Core or MySQL DB? My local setup is:

• .NET Core 2.2.0
• EF Core 2.2.4
• Pomelo.EntityFrameworkCore.MySql 2.2.0

[JonathanMagnan]

Hello @PayteR ,

Thank for reporting.

I believe this option is just not yet supported by our library for Pomelo. It would explain why it works on Fiddle but not in your application.

We will look at it and release a new version early this week that support it.

I will give you an update very soon.

[JonathanMagnan]

Sorry @PayteR for the long waiting,

This issue is currently harder to fix then we initially thought.

I believe we are finally close to a solution

[PayteR]

Thank you for your work, but i have made it already at the lowest level of abstraction by calling LOAD DATA INFILE on temp table and all queries directly to MySQL https://github.com/artiomchi/FlexLabs.Upsert/issues/63