sayn
sayn copied to clipboard
173TECH
Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
Applied cache_size=0 in Environment compiler to force recompilation of sql macros.
I'm running a copy task with a table `users` that contains a `metadata` column with type `json`. ```yaml tasks: copy_users: type: copy source: db: input_db table: users destination: db: output_db...
Integrate the option to initialise different projects: 1. Barebones project (default `sayn init`), this should include only the minimum required to get a SAYN project going from scratch. 2. Example...
`load_data` crashes when records are not homogeneous. We can simply reshape the batch before issuing the insert statement to fix it here: https://github.com/173TECH/sayn/blob/d1cf36b92fad6a1798b57ad80abb22e8386e0e86/sayn/database/__init__.py#L298
**Database** Currently, database errors raise exceptions and so the message is just the error produced by the driver. We can improve this in known cases like autosql and copy tasks...
- Macros in `macros` folder in `sql` folder should automatically be imported and made available in the Jinja environment. - Investigate how to make those macros available through the Python...
Currently, the BigQuery part of the documentation does not have an example of implementation which makes it difficult to use. Also, the credentials_path is mislabeled as required when it's not...
Review the overall errors from Python tasks. We need to be able to differentiate easily errors which are caused to SAYN setup (e.g. missing module file, wrong name of module...
Would be great to have the possibility to work with DB2 databases. It should be fairly easy to implement. https://www.ibm.com/support/knowledgecenter/SSEPGG_11.5.0/com.ibm.swg.im.dbclient.python.doc/doc/t0060891.html
