graylog2-server icon indicating copy to clipboard operation
graylog2-server copied to clipboard
verified publisher icon Graylog2

[bug] Data node does not start up successfully after importing custom CA.

Open mako42 opened this issue 1 year ago • 1 comments

This is the bug Dan experienced and reported in Slack.

Dan used his own windows CA. He had no issues importing it, but afterwards the data node does not start successfully.

Expected Behavior

Current Behavior

The data node does not start successfully after the CA import, instead it throws errors:

Index cds_4 migration failed after 0 seconds: GetTaskResponse[completed=true, task=Task[node=jMDF5RCbRSKzHnzcv4i1eA, id=8234, type=transport, action=indices:data/write/reindex, status=TaskStatus[total=0, updated=0, created=0, deleted=0, batches=0, versionConflicts=0, noops=0, failures=null], description=reindex from [scheme=https host=glos01.eclipsenetwork.org port=9200 pathPrefix=/ query={ "match_all" : { "boost" : 1.0 } } username=elastic password=<<>>][cds_4] to [cds_4], startTimeInMillis=1719421000765, runningTimeInNanos=350832478, cancellable=true, cancelled=false, headers={X-Opaque-Id=667c47ce66e1a566a1b983a9}], error=type='s_s_l_handshake_exception', reason='PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target', causedBy='{type=validator_exception, reason=PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, caused_by={type=sun_cert_path_builder_exception, reason=unable to find valid certification path to requested target}}'].

Seems like the certificate is the culprit, there's a certificate_unknown in its output:

2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] [2024-06-26T18:02:25,448][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [gldn03.lab.eclipsenetwork.org] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?] 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?] 2024-06-26T18:02:25.450Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?] 2024-06-26T18:02:25.451Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:310) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1445) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.452Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.453Z INFO [OpensearchProcessImpl] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] 2024-06-26T18:02:25.454Z INFO [OpensearchProcessImpl] [2024-06-26T18:02:25,450][WARN ][o.o.h.AbstractHttpServerTransport] [gldn03.lab.eclipsenetwork.org] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.99.92:9200, remoteAddress=/192.168.1.231:40352}

Possible Solution

Steps to Reproduce (for bugs)

  1. Start migration (remote reindexing)
  2. Upload own CA
  3. Data node won't start up successfully.

Context

Migration testing.

Your Environment

  • Graylog Version: 6.1 alpha 3
  • Java Version:
  • OpenSearch Version:
  • MongoDB Version:
  • Operating System:
  • Browser version:

mako42 avatar Jun 27 '24 06:06 mako42

Linking @mcdowellster: for any questions and missing info, ask him and not me :D

mako42 avatar Jun 27 '24 06:06 mako42