DataflowTemplate
DataflowTemplate copied to clipboard
mercari
Mercari Dataflow Template
Modify test codes in private repositories that contain data and logic for internal use so that it can be released to the public.
* Update dependencies * Support float32 data type for spanner * Support avro 1.11
## WHAT Adding the `INSERT_OR_UPDATE` mode support for PostgreSQL JDBC sink. ## WHY Currently, the `INSERT_OR_UPDATE` mode is only available for MySQL. Recent versions of PostgreSQL (v15+) support `MERGE` SQL...
## WHAT Include the [com.google.cloud/alloydb-jdbc-connector](https://central.sonatype.com/artifact/com.google.cloud/alloydb-jdbc-connector) package so that `com.google.cloud.alloydb.SocketFactory` can be used. ## WHY To enhance support for AlloyDB.
## WHAT Make the `user` and `password` parameters optional to support IAM user connections using the `enableIamAuth=true` parameter. ## WHY The JDBC source module already includes this feature.
Issue: #101 * add uuid type for avro record * suppor uuid sql type for jdbc source and sink module
## WHAT * Support UUID sql type for jdbc source, sink module. ## WHY * To enhance RDB data transfer.
## Expected Behavior Pipeline with Spanner is launched to Dataflow and executed successfully ## Current Behavior Facing an error when launching any Dataflow which has Spanner as source ot target...
## WHAT Several modules expose an `outputType` parameter, but it isn’t documented in their READMEs. ## WHY For example, the JDBC sink module doesn’t support the `ELEMENT` type, which makes...