streaming-benchmarks icon indicating copy to clipboard operation
streaming-benchmarks copied to clipboard
verified publisher icon yahoo

add gearpump benchmark

Open huafengw opened this issue 10 years ago • 7 comments

Gearpump is an Akka based real-time big data streaming engine initiated by Intel. Before opening this pull request, we did the benchmark tests in our own environment comparing with Storm and here is the brief result:

Cluster setup:

  • 4 nodes in total, each with Intel Xeon processors running at 2.90 GHz, with total of 32 cores. Each node has 64G memory
  • 10 Gb network
  • 10 second windows
  • 100 campaigns
  • 10 ads per campaign
  • 4 Kafka nodes with 4 partitions, not replication
  • 1 Redis node
  • 16 worker nodes (Storm)
  • 3 zookeeper nodes

We launch multiple Kafka producers on each node to create heavy load and we mainly test the load at 170,000 events/s in total.

Storm setup:

  • Storm 0.10.0
  • No ack
  • 4 supervisor nodes with default configuration

Gearpump setup:

  • One Worker per node
  • All default configuration

Comparison result at 170,000 events per second: untitled

Generally our initial result shows that Gearpump's performance is fairly close to Storm's, sometimes Storm is better. Please note that Storm without acking can not have at-least-once guarantee while Gearpump still have that even if with high throughput.

When running sh stream-benchmarks.sh GEARPUMP_TEST , you can browser http:///localhost:8090 to see Gearpump's web UI, the default user name is admin ans the password is admin. You will be able to monitor the running streaming application something like:

capture

Hope you guys can try our project and we are welcome to any question or advice:)

huafengw avatar Jan 28 '16 05:01 huafengw

Nice! I'll take a closer look at this tomorrow.

knusbaum avatar Jan 28 '16 05:01 knusbaum

Thank you for submitting this pull request, however I do not see a valid CLA on file for you. Before we can merge this request please visit https://yahoocla.herokuapp.com/ and agree to the terms. Thanks! :smile:

yahoocla avatar Jan 28 '16 05:01 yahoocla

CLA is valid!

yahoocla avatar Jan 28 '16 05:01 yahoocla

Sorry this took so long. We are trying to fix things so that community participation is better taken care of.

The code looks great, but when we go to download the release, we get an HTML page, not the actual gearpump release, with a redirection link to AWS. Could you take a look at that please?

If I manually download the release everything works fine so I am +1 after this is fixed.

revans2 avatar Jul 11 '16 17:07 revans2

Sorry about that, I'll let you know after I correct the code.

huafengw avatar Jul 12 '16 02:07 huafengw

@huafengw it is not a big deal, spark changed things on us too and the build does not work out of the box for spark so I am working on a patch for that too.

revans2 avatar Jul 12 '16 14:07 revans2

@revans2 Sorry for the late response. I just update the pull request and hope everything works fine.

huafengw avatar Oct 14 '16 05:10 huafengw