dbt_fivetran_utils icon indicating copy to clipboard operation
dbt_fivetran_utils copied to clipboard

add fivetran_union_relations macro

Open fivetran-jamie opened this issue 2 years ago • 0 comments

What change does this PR introduce?

Adds fivetran_union_relations macro. This was necessary for rolling union_data out to Zendesk, which has some source tables that have reserved-keywords for names (ie timezone). In the past, we've solved this error by aliasing the problematic source table, but there's no way to get dbt_utils.union_relations to do so (besides opening a PR on dbt utils or making our own version of the macro).

We already do have our own version of union_relations in this package, but as we learned recently, using the same exact name as a macro in debt_utils can cause failures for customers with certain dispatch configs

This PR also adjusts union_data (in order to work with Netsuite, which is a weird package due to our support of both of its endpoints).

  • new optional (none by default) connector_table_name_override field. this is to be used when the defined source table name is different from the actual table name as it appears in the connector/warehouse. This is only a thing for Netsuite2
    • Added a reference to a new global variable use_table_name_identifer_override. this was necessary to add for our integration tests (and customers) to still be able to use identifier variables for tables in which connector_table_name_override is implemented.

If this PR introduces a new macro, how did you test the new macro?

Ran it with zendesk, hubspot, and netsuite in their feature/add-union-data branches. moreover, i tested this fivetran_utils branch on Shopify_source, which has the union ability already.

If this PR introduces a modification to an existing macro, which packages is the macro currently present in and what steps were taken to test compatibility across packages?

  • just added a comment to the union_relations macro to deprecate it in the future

Ran it with zendesk, hubspot, and netsuite in their feature/add-union-data branches. moreover, i tested this fivetran_utils branch on Shopify_source, which has the union ability already.

Did you update the README to reflect the macro addition/modifications?

  • [x] Yes
  • [ ] No (provide further explanation)

fivetran-jamie avatar Dec 26 '23 23:12 fivetran-jamie