summingbird hbase scalding Store based on maple/storehaus

Dec 10 '13 21:12 joshby247

Thanks for working on this!

Dec 11 '13 00:12 johnynek

I've incorporated the feedback. I removed the error check for the empty pipe in readLast. It looks like ScaldingStore's merge should handle this correctly

Dec 12 '13 23:12 joshby247

in addition to using a converted HBaseByteArrayStore I also renamed main() in ScaldingRunning so the "getting started" wiki instructions weren't impacted by another main() in summingbird-example

Dec 13 '13 19:12 joshby247

any update on this PR. Would love to get this merged in as I have to use HBase with SB too

Dec 27 '13 00:12 MansurAshraf

Sorry for the slow response. @MansurAshraf I guess you have reviewed the HBase stuff. We don't use it much at Twitter, so I'm only giving a summingbird review.

I don't see how this code is tracking the state of which batches this store has completed. That information needs to be available at planning time, and this code is currently just always claiming to have data. That will not be correct.

That said, we do need to build some kind of framework to make it easier to test Stores.

Dec 29 '13 19:12 johnynek

Added stuff to write/read the last processed BatchID from ZK. I have to confess that I don't have much experience interacting with ZK directly (mostly use systems that use ZK) so I'm quite open to feedback on how to do that better. It does seem like the ideal way to do this would be to register a Watcher with the yet-to-be-created zookeeper WaitingState so the BatchID could be written once the Scalding job has completed successfully.

Jan 09 '14 21:01 joshby247

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

Jul 18 '19 15:07 CLAassistant