danieljoos
"Topic or partition does not exist on this broker" error
Hi - We've been testing libkafka-asio for a while and it has been great. However, we run into this error once in a while and haven't been able to figure out what is causing it:
Topic or partition does not exist on this broker
The topic in question clearly exists on the broker, and sometimes restarting the Kafka service resolves the issue. Has anyone else run into this problem? I'm not sure if it is a client or server issue.
Thanks! Mike
Hi Mike, According to the description in the Kafka source, the error can either be client-side (missing/old metadata) or server side (topic/partition does not exist). Here's the pointer: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/common/UnknownTopicOrPartitionException.scala#L20
So maybe a refresh of the client's metadata already helps here. Maybe another broker is now the leader for this partiton (personally, I would expect another error in such a case ...).
Thanks for the reply! Is there a way to trigger a refresh of the client's metadata? In my particular setup, I only have 1 broker in the system. It has been difficult to consistently reproduce this issue, but some clients will start getting these error messages, while other clients seem to be fine.
Are you using Kafka 0.9.0.0 as server?Might be a server related bug. Have seen the same kind of strange things with the new kafka versions and librdkafka.Had no time to switch to this implementation yet :(I installed Kafka 0.9.0.1 and have not seen it since - but was not testing so much lately. Von meinem Samsung Gerät gesendet.
-------- Ursprüngliche Nachricht -------- Von: mtolman [email protected] Datum: 21.03.2016 19:29 (GMT+01:00) An: danieljoos/libkafka-asio [email protected] Betreff: Re: [libkafka-asio] "Topic or partition does not exist on this broker" error (#18)
Thanks for the reply! Is there a way to trigger a refresh of the client's metadata? In my particular setup, I only have 1 broker in the system. It has been difficult to consistently reproduce this issue, but some client will start getting these error messages, while other clients seem to be fine.
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub
Are you using Kafka 0.9.0.0 as server?Might be a server related bug
I think you might be right. I'm, using another C++ asio client and I had to fix some things in the library when we moved to 0.9.0 and I remember errors like this... - we have not yet moved to 0.9.0.1
2016-03-21 21:01 GMT+01:00 A. Binzxxxxxx [email protected]:
Are you using Kafka 0.9.0.0 as server?Might be a server related bug. Have seen the same kind of strange things with the new kafka versions and librdkafka.Had no time to switch to this implementation yet :(I installed Kafka 0.9.0.1 and have not seen it since - but was not testing so much lately. Von meinem Samsung Gerät gesendet.
-------- Ursprüngliche Nachricht -------- Von: mtolman [email protected] Datum: 21.03.2016 19:29 (GMT+01:00) An: danieljoos/libkafka-asio [email protected] Betreff: Re: [libkafka-asio] "Topic or partition does not exist on this broker" error (#18)
Thanks for the reply! Is there a way to trigger a refresh of the client's metadata? In my particular setup, I only have 1 broker in the system. It has been difficult to consistently reproduce this issue, but some client will start getting these error messages, while other clients seem to be fine.
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub
— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/danieljoos/libkafka-asio/issues/18#issuecomment-199450181
No - we're using 0.8.2.2 as the server. We are running on Windows - hopefully that isn't causing issues :)
If you are speaking windows client then i do this all the time for development and that should not cause any issues (with another library - but there should not be any architectural differences)
2016-03-21 23:03 GMT+01:00 mtolman [email protected]:
No - we're using 0.8.2.2 as the server. We are running on Windows - hopefully that isn't causing issues :)
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/danieljoos/libkafka-asio/issues/18#issuecomment-199507587
never tried the server. :-)
2016-03-21 23:07 GMT+01:00 mtolman [email protected]:
We're running on Windows for client and server.
— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/danieljoos/libkafka-asio/issues/18#issuecomment-199509283
Was your client connecting before the broker finished startup? link
Was the broker going to shutdown?
Is a restart of the broker solving this? (It was for me when I had this - had 3brokers per cluster)
May the topic really not exist at first but then get created by the broker (topic auto create)?
Yes, restarting Kafka usually solves the issue. I've been researching this issue the last couple of days, and it appears that it is related to Kafka not shutting down correctly. For example, if I kill the Kafka process (through task manager in Windows), the issue occurs quite often. I'm wondering if there is some manual cleanup that needs to happen if the process doesn't shut down correctly?
