feat: add GCS Avro to Spanner using SCD

[nitobuendia]

Pipeline that allows to write data from Avro using Slow Changing Dimensions (SCD) Types 1 and 2, with possibility to expand to others in the future.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 76.50273% with 129 lines in your changes missing coverage. Please review.

Project coverage is 43.38%. Comparing base (bbf744f) to head (cb5391f). Report is 215 commits behind head on main.

Files with missing lines	Patch %	Lines
...eleport/v2/templates/AvroToSpannerScdPipeline.java	26.36%	73 Missing and 8 partials :warning:
...ort/v2/transforms/SpannerScdMutationTransform.java	0.00%	22 Missing :warning:
...ogle/cloud/teleport/v2/utils/StructComparator.java	78.72%	8 Missing and 2 partials :warning:
...loud/teleport/v2/utils/CurrentTimestampGetter.java	0.00%	6 Missing :warning:
...teleport/v2/transforms/SpannerScdMutationDoFn.java	95.69%	2 Missing and 2 partials :warning:
...google/cloud/teleport/v2/utils/SpannerFactory.java	94.11%	1 Missing and 2 partials :warning:
...m/google/cloud/teleport/v2/utils/StructHelper.java	98.01%	1 Missing and 1 partial :warning:
...e/cloud/teleport/v2/transforms/AvroToStructFn.java	98.76%	0 Missing and 1 partial :warning:

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1795      +/-   ##
============================================
+ Coverage     42.58%   43.38%   +0.80%     
- Complexity     3364     3485     +121     
============================================
  Files           815      828      +13     
  Lines         47672    48475     +803     
  Branches       5107     5221     +114     
============================================
+ Hits          20301    21031     +730     
- Misses        25717    25754      +37     
- Partials       1654     1690      +36     

Components	Coverage Δ
spanner-templates	64.60% <ø> (+0.03%)	:arrow_up:
spanner-import-export	63.88% <ø> (-0.57%)	:arrow_down:
spanner-live-forward-migration	75.29% <ø> (+0.28%)	:arrow_up:
spanner-live-reverse-replication	52.44% <ø> (+0.48%)	:arrow_up:
spanner-bulk-migration	83.75% <ø> (+0.30%)	:arrow_up:

Files with missing lines	Coverage Δ
...d/teleport/v2/transforms/MakeBatchesTransform.java	100.00% <100.00%> (ø)
...e/cloud/teleport/v2/transforms/AvroToStructFn.java	98.76% <98.76%> (ø)
...m/google/cloud/teleport/v2/utils/StructHelper.java	98.01% <98.01%> (ø)
...google/cloud/teleport/v2/utils/SpannerFactory.java	94.11% <94.11%> (ø)
...teleport/v2/transforms/SpannerScdMutationDoFn.java	95.69% <95.69%> (ø)
...loud/teleport/v2/utils/CurrentTimestampGetter.java	0.00% <0.00%> (ø)
...ogle/cloud/teleport/v2/utils/StructComparator.java	78.72% <78.72%> (ø)
...ort/v2/transforms/SpannerScdMutationTransform.java	0.00% <0.00%> (ø)
...eleport/v2/templates/AvroToSpannerScdPipeline.java	26.36% <26.36%> (ø)

... and 51 files with indirect coverage changes

[nitobuendia]

The CL is ready for review.

I see the IT tests failing, not just for my pipeline but for all Spanner related pipelines. However, the IT tests work when triggered locally. Is this a permission issue or something else going on? Happy to make required changes, but unsure how to go about it.

Thanks.

[nitobuendia]

This template has been published under a different GoogleCloudPlatform repository. As a result, I will close this pull request.

If you are reading this and are interested in loading Apache Avro files to Cloud Spanner with Slowly Changing Dimensions (SCD) - or to regularly move data to Spanner after the initial migration - you can get this template on this repository or you can follow this guide.