V2 client query fails on initial attempt

[maxsumrall]

Describe the bug

It appears that the first queries (SELECTs) to the ClickHouse server fail on the V2 client with an error: com.clickhouse.client.api.ClientException: Failed to get query response. Wrapping the request in a retry loop seems to resolve after the first failed attempt.

Steps to reproduce

1. Start up application and connect to ClickHouse Cloud. This happens on 24.5, but has not been observed with 24.6 It has not been observed locally.
2. Execute query:

client.queryAll(QUERY).stream()
        .map(record -> ...record mapping...)
        .collect(toImmutableList());

Expected behaviour

Query succeeds.

Code example

See above.

Error log

com.clickhouse.client.api.ClientException: Failed to get query response
	at com.clickhouse.client.api.Client.queryAll(Client.java:1483)
	...redacted...
Caused by: com.clickhouse.client.api.ClientException: Failed to get query response
	at com.clickhouse.client.api.Client.lambda$query$10(Client.java:1409)
	at com.clickhouse.client.api.Client.runAsyncOperation(Client.java:1624)
	at com.clickhouse.client.api.Client.query(Client.java:1414)
	at com.clickhouse.client.api.Client.query(Client.java:1284)
	at com.clickhouse.client.api.Client.queryAll(Client.java:1467)
	... 21 common frames omitted
Caused by: java.util.concurrent.CompletionException: com.clickhouse.client.ClickHouseException: Connection pool shut down
	at com.clickhouse.client.ClickHouseClientBuilder$Agent.retry(ClickHouseClientBuilder.java:240)
	at com.clickhouse.client.ClickHouseClientBuilder$Agent.handle(ClickHouseClientBuilder.java:269)
	at com.clickhouse.client.ClickHouseClientBuilder$Agent.send(ClickHouseClientBuilder.java:296)
	at com.clickhouse.client.ClickHouseClientBuilder$Agent.execute(ClickHouseClientBuilder.java:349)
	at com.clickhouse.client.ClickHouseRequest.execute(ClickHouseRequest.java:2144)
	at com.clickhouse.client.api.Client.lambda$query$10(Client.java:1400)
	... 25 common frames omitted
Caused by: com.clickhouse.client.ClickHouseException: Connection pool shut down
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:151)
	at com.clickhouse.client.ClickHouseClientBuilder$Agent.handle(ClickHouseClientBuilder.java:251)
	... 29 common frames omitted
Caused by: java.lang.IllegalStateException: Connection pool shut down
	at org.apache.hc.core5.util.Asserts.check(Asserts.java:38)
	at org.apache.hc.core5.pool.LaxConnPool.lease(LaxConnPool.java:163)
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.lease(PoolingHttpClientConnectionManager.java:297)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.acquireEndpoint(InternalExecRuntime.java:103)
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:125)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:113)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116)
	at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
	at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
	at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:123)
	at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:280)
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:195)
	at com.clickhouse.client.AbstractClient.execute(AbstractClient.java:280)
	at com.clickhouse.client.ClickHouseClientBuilder$Agent.sendOnce(ClickHouseClientBuilder.java:282)
	at com.clickhouse.client.ClickHouseClientBuilder$Agent.send(ClickHouseClientBuilder.java:294)
	... 28 common frames omitted

Configuration

Environment

• Client version: 0.6.5
• Language version: 21.0.2
• OS: debian

ClickHouse server

• ClickHouse Server version: 24.5

[chernser]

Good day, @maxsumrall ! Thank you for reporting the issue!

Do you close client at any case? The error indicates that underlying client was closed.

Would you please try configuring client with .useNewImplementation(true) - it will switch client to use new HTTP transport layer that we recently tested and it should not have such issue.

Is the remote server Cloud instance? Is there a proxy between client and the server?

Thanks!

[maxsumrall]

Would you please try configuring client with .useNewImplementation(true) - it will switch client to use new HTTP transport layer that we recently tested and it should not have such issue.

Sure, I'll investigate using it. Is there any docs about the new and old implementation is? N.B. This is marked in the code as: @deprecated - do not use - it is only for development

Is the remote server Cloud instance? Is there a proxy between client and the server?

Yes using CH cloud.

[chernser]

@maxsumrall Initially we have created new client v2 API and had to use old client for testing and trying out the new API. While that time we have developed a new transport layer what is called a "newImplementation". In the future we will remove usage of an old client to deprecate it.