InvalidCookieException

[jihwong]

BUG REPORT

Describe the bug

Error reported after restarting the bookkeeper node. Due to being a Kubernetes environment, it was successfully started after being restarted x times. Why is there a new instanceId df26468b-558a-41fc-8705-3d7dc21d2817

log

2023-12-12T10:03:04,513 - INFO  - [main-EventThread:ZooKeeperWatcherBase@130] - ZooKeeper client is connected now.
2023-12-12T10:03:04,650 - ERROR - [main:Main@218] - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: instanceId df26468b-558a-41fc-8705-3d7dc21d2817 is not matching with 67c327e1-d3d6-4e28-b731-85a9ae63920b
	at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:168) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]
	at org.apache.bookkeeper.bookie.Cookie.verify(Cookie.java:173) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]
	at org.apache.bookkeeper.bookie.LegacyCookieValidation.verifyAndGetMissingDirs(LegacyCookieValidation.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]
	at org.apache.bookkeeper.bookie.LegacyCookieValidation.checkCookies(LegacyCookieValidation.java:84) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]
	at org.apache.bookkeeper.server.EmbeddedServer$Builder.build(EmbeddedServer.java:408) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:277) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]
	at org.apache.bookkeeper.server.Main.doMain(Main.java:216) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]
	at org.apache.bookkeeper.server.Main.main(Main.java:199) ~[org.apache.bookkeeper-bookkeeper-server-4.16.2.jar:4.16.2]

root@ubuntu:~# k get pod -n pulsar 
NAME                                     READY   STATUS      RESTARTS      AGE
bookie-sts-0                             1/1     Running     0             16h
bookie-sts-1                             1/1     Running     5 (16h ago)   16h
bookie-sts-2                             1/1     Running     0             16h

zookeeper

[zk: localhost:2181(CONNECTED) 117] get /ledgers/cookies/bookie-sts-1.bookie-svc.pulsar.svc.cluster.local:3181
5
bookieHost: "bookie-sts-1.bookie-svc.pulsar.svc.cluster.local:3181"
journalDir: "/bookkeeper/data/journal"
ledgerDirs: "1\t/bookkeeper/data/ledgers"
instanceId: "67c327e1-d3d6-4e28-b731-85a9ae63920b"
indexDirs: "1\t/bookkeeper/data/ledgers"

data dir

root@bookie-sts-0:/bookkeeper/data# cat ledgers/current/VERSION 
5
bookieHost: "bookie-sts-0.bookie-svc.pulsar.svc.cluster.local:3181"
journalDir: "/bookkeeper/data/journal"
ledgerDirs: "1\t/bookkeeper/data/ledgers"
instanceId: "67c327e1-d3d6-4e28-b731-85a9ae63920b"
indexDirs: "1\t/bookkeeper/data/ledgers"
root@bookie-sts-0:/bookkeeper/data# cat journal/current/VERSION 
5
bookieHost: "bookie-sts-0.bookie-svc.pulsar.svc.cluster.local:3181"
journalDir: "/bookkeeper/data/journal"
ledgerDirs: "1\t/bookkeeper/data/ledgers"
instanceId: "67c327e1-d3d6-4e28-b731-85a9ae63920b"
indexDirs: "1\t/bookkeeper/data/ledgers"

To Reproduce

Steps to reproduce the behavior:

1. Restart the bookkeeper node

Expected behavior

Screenshots

Additional context

1. kubernetes environment
2. image: apache/bookkeeper:4.16.2

[jihwong]

I configured useHostNameAsBookieID=true, but it seems to be unrelated to the issue.

[zwOvO]

Assign a separate /bookkeeper/data directory for each k8s stat

such： /bookkeeper/bookie-sts-0-data /bookkeeper/bookie-sts-1-data /bookkeeper/bookie-sts-2-data

[Ghatage]

AFAIK InstanceId is only set when a new cluster is instantiated, not during restarts.

So you either have an installation which has an existing instance id in the registration manager, or your k8s init command is attempting to reinitialize the cluster.

Can you please check the logs for this line Initializing ZooKeeper metadata for new cluster. It should indicate a new cluster id is being set.

cc @eolivelli

[hezhangjian]

closed by no updates, feel free to reopen it if you have new informations or way to reproduce.