createCluster is stuck without throwing any error with TLS mode

[prameet-verma]

Description

I am running my NextJS application which uses AWS Elasticahe Redis Cluster on server side. Now I am trying use a different redis cluster but in TLS mode with these settings:

This is my cluster connection code:

    console.log("🚀 ~ file: index.ts:58 ~ redisConfig:", redisConfig)
    const NAMESPACE = redisConfig.expCachePrefix;
    const cluster = createCluster({
      rootNodes: [
        {
          socket: {
            host: redisConfig.host,
            port: redisConfig.port,
            tls: true,
            checkServerIdentity: () => undefined,
          },
        },
      ],
    });
    console.log("🚀 ~ file: index.ts:72 ~ cluster created:")
   
    cluster.on('error', (err) => {
      console.log('Redis Cluster Error', err);
      throw err;
    });

    await cluster.connect(); // This will throw error if unsuccessful
    console.log("🚀 ~ file: index.ts:83 ~ below cluster connect")
    await cluster.set("appKey","appVaue");
    console.log("🚀 ~ file: index.ts:85 ~ below cluster set command:")
    cluster.on('ready', () => console.log("Cluster is ready now"))
    cluster.on('error', (err) => debugLog('Redis Cluster Error', err));

In my console I see log till "🚀 ~ file: index.ts:83 ~ below cluster connect" only, the client is stuck at await cluster.set("appKey","appVaue"); not moving ahead leading to block the application from moving ahead and serve requests.

The interesting part is that I am able to connect to redis and perform set and get operations via cli.

Please note that my VPN is already whitelisted in the security group of redis cluster.

Node.js Version

v16.20.1

Redis Server Version

No response

Node Redis Version

4.4.0

Platform

macOS

Logs

No response

[leibale]

Whatever you set in rootNodes[*] will be used only for that specific node, while what you set in defaults will be used for every node in the cluster. You should move the tls: true to the defaults to make all the sockets use TLS:

const cluster = createCluster({
  rootNodes: [{
    socket: {
      host: redisConfig.host,
      port: redisConfig.port
    }
  }],
  defaults: {
    tls: true,
    checkServerIdentity: () => undefined // do you actually need that?
  }
});

[prameet-verma]

Whatever you set in rootNodes[*] will be used only for that specific node, while what you set in defaults will be used for every node in the cluster. You should move the tls: true to the defaults to make all the sockets use TLS:

const cluster = createCluster({
  rootNodes: [{
    socket: {
      host: redisConfig.host,
      port: redisConfig.port
    }
  }],
  defaults: {
    tls: true,
    checkServerIdentity: () => undefined // do you actually need that?
  }
});

checkServerIdentity: () => undefined // do you actually need that? I am using this for my local machine only as I am not passing certificates.

I tried your suggestion as well and even that's not working @leibale

[leibale]

When you connect to one of the nodes and run CLUSTER NODES, do you get local or remote addresses? Cause if it's local addresses, use "node address map" to map between the local addresses and the remote ondes

[dinesh196]

When you connect to one of the nodes and run CLUSTER NODES, do you get local or remote addresses?

we are using remote address for the host url.

[leibale]

1. Run CLUSTER NODES on one of the nodes in the cluster
2. Copy the host:port part (the 2 column in every row)
3. Try to connect to them using redis-cli (from the same machine you run the node script) Did it work? are you able to connect to all of them?

[prameet-verma]

1. Run CLUSTER NODES on one of the nodes in the cluster
2. Copy the host:port part (the 2 column in every row)
3. Try to connect to them using redis-cli (from the same machine you run the node script) Did it work? are you able to connect to all of them?

Tried this as well with the below code but no lead:

import { createCluster } from 'redis';
const cluster = createCluster({
  rootNodes: [
    {
      //   url:"rediss://clustercfurl:6379",
      // tlsConfig:{
      //     checkServerIdentity: () => undefined,
      // }
      socket: {
        host: 'cache-0001-001(with node url)',
        port: 6379,
        tls: true,
        checkServerIdentity: () => undefined,
      },
    },
    {
      socket: {
        host: 'cache-0002-001',
        port: 6379,
        tls: true,
        checkServerIdentity: () => undefined,
      },
    },
    {
      socket: {
        host: 'cache-0001-002',
        port: 6379,
        tls: true,
        checkServerIdentity: () => undefined,
      },
    },
    {
      socket: {
        host: 'cache-0001-003',
        port: 6379,
        tls: true,
        checkServerIdentity: () => undefined,
      },
    },
    {
      socket: {
        host: 'cache-0002-002',
        port: 6379,
        tls: true,
        checkServerIdentity: () => undefined,
      },
    },
    {
      socket: {
        host: 'cache-0002-003',
        port: 6379,
        tls: true,
        checkServerIdentity: () => undefined,
      },
    },
  ],
//   defaults: {
//     tls: true,
//     checkServerIdentity: () => undefined, // do you actually need that?
//   },
});

console.log('🚀 ~ file: index.ts:72 ~ cluster created:');

cluster.on('error', (err) => {
  console.log('Redis Cluster Error', err);
  throw err;
});

cluster
  .connect()
  .then((c) => {
    console.log('Cluster Connected', c);
    cluster
      .set('node-redis', 'redis-value')
      .then(() => console.log('Cluster value set'))
      .then(() => console.log('value set succesfully'))
      .catch((err) => console.log('Set error', err));
  })
  .catch((err) => console.log(err));

// This will throw error if unsuccessful
console.log('🚀 ~ file: index.ts:83 ~ below cluster connect');

// cluster.set('value', 'value').then(() => console.log('Cluster value set'));
// await cluster.set("appKey","appVaue");
console.log('🚀 ~ file: index.ts:85 ~ below cluster set command:');
cluster.on('ready', () => console.log('Cluster is ready now'));
cluster.on('error', (err) => console.error('Redis Cluster Error', err));

Output:

In my Elasticache config, I have 2shards and 3 nodes per shard.

Also, I am able to access (both get and set) via redis-cli. redis-cli --tls --cluster call clusterURL:6379 keys "*" @leibale

[prameet-verma]

The client is able to make connection to the redis cluster but not able to perform any get or set operation. I am able to get the master nodes by running this cluster.masters

[leibale]

@prameet-verma Anything that should be on all the sockets opened by the cluster should be in defaults, not in rootNodes (so you should uncomment defaults and remove tls and checkServerIdentity from every root node.

I'm actually not exactly sure whats going on here, but I'm pretty sure that one of the nodes is unreachable..

can you loop over cluster.masters and check if the clients are connected?

[prameet-verma]

Would you mind hopping on a call? @leibale

[prameet-verma]

One more interesting observation is that the below code works perfectly fine with TLS disabled in cluster mode.

const cluster = createCluster({
  rootNodes: [{
    socket: {
      host: redisConfig.host,
      port: redisConfig.port
    }
  }],

});

[prameet-verma]

@prameet-verma Anything that should be on all the sockets opened by the cluster should be in defaults, not in rootNodes (so you should uncomment defaults and remove tls and checkServerIdentity from every root node.

I'm actually not exactly sure whats going on here, but I'm pretty sure that one of the nodes is unreachable..

can you loop over cluster.masters and check if the clients are connected?

I tried this as well and now it's not able to connect to the cluster as well. Setting values in defaults in not even able to connect

[leibale]

@prameet-verma I'm in the middle of doing some things, we can have a call next week if u want :)

[leibale]

@prameet-verma I actually have 20-30 min now, send me a meet/zoom/whatever link here if you got time..

[prameet-verma]

Sorry @leibale , I am in IST timezone and was up till 1am only. We can hop on call now, if you are available or I can book your calendar (please share email).

[lnwu]

is there any way to using Configuration endpoint from aws instead of config each node for cluster mode?

[deblanco]

Hey same here, how did this end up? @lnwu @prameet-verma

[lnwu]

@deblanco we move to ioredis