Brook Elgie

Ah, okay. So I yield more than one pipeline spec from the generator? That makes sense. One downside is the Generator class might become a bit monolithic.

I found I needed a passing understanding of data packages to understand how to write pipelines and processors.

DPP isn't just moving data from one point to another, but also transforming, changing and filtering it. Not sure how that helps, but perhaps it's more of an assembly line,...

Similar issue being experienced with service. From the [logs](https://console.cloud.google.com/logs/viewer?interval=PT1H&project=oki-cloud&minLogLevel=0&expandAll=false&timestamp=2019-02-25T11:51:10.060000000Z&customFacets=&limitCustomFacetWidth=true&scrollTimestamp=2019-02-25T11:49:08.000000000Z&advancedFilter=resource.type%3D%22container%22%0Aresource.labels.cluster_name%3D%22oki%22%0Aresource.labels.namespace_id%3D%22production%22%0Aresource.labels.project_id%3D%22oki-cloud%22%0Aresource.labels.zone:%22europe-west1-b%22%0Aresource.labels.container_name%3D%22goodtables-worker%22&dateRangeUnbound=forwardInTime&dateRangeStart=2019-02-25T10:51:10.060Z&dateRangeEnd=2019-02-25T11:51:10.060Z): ```[2019-02-25 11:49:08,161: ERROR/MainProcess] Task handler raised error: StatementError("(sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back",)```

Pod restarted on k8s infrastructure, as per command above, and working again for now.

Also in the user's activity stream `/user/activity/`.

Great to see some work on this. We (Open Knowledge) have also been working on an EZID based DOI extension for datasets: https://github.com/okfn/ckanext-doi . It provides an additional form field...

Anything in [csvkit](https://csvkit.readthedocs.org/) that can help?