data-transfer-project icon indicating copy to clipboard operation
data-transfer-project copied to clipboard

Split JobStore into 2 stores - one for job metadata and one for temp data

Open rtannenbaum opened this issue 7 years ago • 0 comments

As discussed in today's maintainer's meeting, JobStore now serves 2 different purposes:

(1) Job metadata - some of this data is sensitive (user credentials) and some may be less so (job state). values tend to be small so a key-value store works well.

(2) Job temp data - data like photos that is too large to keep in memory of a transfer worker during a job. This data is all sensitive (user PII data) and is often larger, and a blobby store works better than key-value.

Currently various implementations of JobStore are each talking to different backend stores depending on the data passed in. It would be simpler to split up these stores at the core framework layer, into a JobStore and a TempDataStore, and the framework could delegate to either store.

Then extensions may implement TempDataStore as a blobby store. Perhaps encryption (with the worker public key) of all data may make sense to build into the common interface, as opposed to in each implementation. We could use an encryption scheme option like we currently do for credentials.

rtannenbaum avatar Nov 26 '18 18:11 rtannenbaum