Benchmarks should be more accurate, more attention to details please

[loyd]

Thanks for benchmarking.

I've noticed that not all libraries are in the same condition. For instance, loyd/clickhouse.rs uses LZ4 by default, that's a reasonable default, but the best library uses no compression by default AFAIK. Disabling it increases performance x2 on my machine.

I would like to see more attention to such details for all other libraries, not only for your favorite one.

[loyd]

Although, I understand that the main reason for the benchmark is to make loud statements:

Column-oriented design that operates directly with blocks of data Dramatically more efficient Up to 100x faster than row-first design around SQL Up to 700x faster than HTTP API

I agree with them, however:

• Row-first design vs column-first design for cases where only one column is fetched is an incorrect statement. So, this needs to be more correct proof of the efficiency of column-based design. I haven't expected the difference in this benchmark related to this specific point.
• RowBinary over HTTP can easily be much slower than Native, but it's not only about client-library performance but also the implementation of HTTP in CH.
• Why 700x? According to your benchmarks the nearest HTTP library is clickhouse-client (Java, HTTP) | 6.4s | 121M | 16x, only 16x. curl (C, HTTP) | 3.7s | 10M | 9x is even faster. So, why 700x?

Also, It's not only about HTTP or row/column-based design, but also block-style/callback-based design, because awaiting on every row is quite expensive according to perf. It's a good point to think about providing block-style (or callback-based) API in libraries (at least, in Rust).

Also, another note about HTTP vs batched Native: in Native format chunks always contain the finite number of rows, while in the case of HTTP chunks, a row can be split into several chunks, which adds an overhead of additional checks during deserialization.