kafka-websocket icon indicating copy to clipboard operation
kafka-websocket copied to clipboard

high cpu for low traffic topics

Open davidbirdsong opened this issue 11 years ago • 0 comments

Love the websocket gateway. It's allowing us to make use of kafka across the open internet protected in a TLS tunnel.

I'm seeing pretty high CPU utilization for a topic with very, very low message rates ( < 1msg/sec.) Our setup is that we have ~70 clients all in their own consumer group since we want to broadcast messages. I have a feeling the cpu is due to the MaxWait of 100ms and having so many consumer groups.

TRACE 22:54:55,331 [ConsumerFetcherThread-imgix-worker-0062_svab3-1408575091046-704ad865-0-104], issuing to broker 104 of fetch request Name: FetchRequest; Version: 0; CorrelationId: 2028; ClientId: imgix-worker-0062-ConsumerFetcherThread-imgix-worker-0062_svab3-1408575091046-704ad865-0-104; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 bytes; RequestInfo: [purge.origin,0] -> PartitionFetchInfo(496,1048576)

Could you expose these consumer config params to the command line or through the URL?

davidbirdsong avatar Aug 20 '14 23:08 davidbirdsong