custom client implementation guide

[shaunc]

I have combinatorial optimization code in go designed to consume flatbuffers models. I'm researching how I can integrate it into a larger platform, hosted in kubernetes, so as to help with autoscaling/distribution, preprocessing, metadata, project management, etc. Kubeflow looks promising. Graphpipe looks like it might be a natural fit for storage and distribution of training data, but I'm not sure where to look to see how to consume. The workflow I would like is:

1. Load raw data from flat files or DB
2. Clean/transform/visualize data using kubeflow/jupyter, track versions, etc.
3. Select set of feature generation functions, which store data into graphpipe
4. train
5. save & serve resulting model (hopefully back in Kubeflow)

Training isn't necessarily a good fit for tensor flow, as I need to distribute subsets based on boolean conditions (or mask sets) to training nodes I'd prefer to keep it all in flatbuffers, and cache on nodes using local kubernetes volumes.

Is graphpipe a good fit for this project? Where should I look for the information I need to write clients?