ClickHouse.Client icon indicating copy to clipboard operation
ClickHouse.Client copied to clipboard

add way to know what record exception in bulk copy

Open mbtolou opened this issue 4 years ago • 2 comments

Hello,

Can you please add a way to know in exactly which record or row of data an exception is raised in the following methods:

ClickHouseBulkCopy - >WriteToServerAsync

https://github.com/DarkWanderer/ClickHouse.Client/blob/e99d477c27b8f3e03f4a8fb3091309bba621daa1/ClickHouse.Client/Copy/ClickHouseBulkCopy.cs#L173

Thanks.

mbtolou avatar Dec 05 '21 07:12 mbtolou

Hi,

If you are able to reproduce the issue under debugger, you should be able to view the row and i values under "Locals" or "Autos"

DarkWanderer avatar Dec 22 '21 22:12 DarkWanderer

thanks. I rewrote the 'PushBatch' function I created an event for the error mode that returns the corresponding record of the error I also created a test for the above mode

Do you confirm the code?

https://github.com/DarkWanderer/ClickHouse.Client/pull/112

using (var gzipStream = new BufferedStream(new GZipStream(stream, CompressionLevel.Fastest, true), 256 * 1024))
{
    if (useInlineQuery)
    {
        using var textWriter = new StreamWriter(gzipStream, Encoding.UTF8, 4 * 1024, true);
        textWriter.WriteLine(query);
        query = null; // Query was already written to POST body
    }

    using var writer = new ExtendedBinaryWriter(gzipStream);
    object[] excp_rows = null;

    try
    {
        foreach (var row in rows)
        {
            excp_rows = row;
            for (var i = 0; i < row.Length; i++)
            {
                columnTypes[i].Write(writer, row[i]);
            }
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.ToString());
        OnBulkCopyException(this, excp_rows);
    }
}

mbtolou avatar Dec 25 '21 04:12 mbtolou

I've added a special type of exception for this - https://github.com/DarkWanderer/ClickHouse.Client/pull/190. Does this cover your needs?

DarkWanderer avatar Sep 01 '22 16:09 DarkWanderer

I checked it. It is very useful. Thanks

mbtolou avatar Sep 01 '22 17:09 mbtolou