scio Support for streaming to Memcached

When the total size of output data remains small, we can make their reads faster if they're in Memcached. Thoughts are that we could implement Scio's AsyncDoLookupFn to use the Memcached client.

Feb 04 '19 15:02 jshph

@smerdy Hi! can you expand with a small example what are you trying to achieve?

Feb 19 '19 16:02 regadas

@smerdy - ping on some more clarification.

Mar 15 '19 14:03 jbx

Hi @jbx @regadas, we were intending to stream Natural Language Query logs (30-40 rps on average) to something with quick reads from a backend service on small data. The data would also have a short TTL. More info on our use case here: https://docs.google.com/document/d/1PBw9Fe-D0Fd7DoEs3yon1HI8AlYoc0MNuVdjpVY3OV4/edit?ts=5c4a0627#heading=h.uga21h7hk24x

We ended up going with BIgTable given the time constraints, which hasn't given us any issues so far. (BigTable instance) But there's some sense that an in-memory store would be ideal.

Mar 19 '19 14:03 jshph

We actually have an memcached sink internally. Should probably clean it up and open source.

Mar 21 '19 18:03 nevillelyh