Standardize unioning macros to use fivetran_utils versions

.gitignore

@@ -61,6 +61,7 @@ service-account.json
 .cache/
 *.temp
 *.tmp
+worktrees/
 
 # Virtual environments
 .conda/

CHANGELOG.md

@@ -1,3 +1,11 @@
+# dbt_marketo v1.4.0
+
+[PR #57](https://github.com/fivetran/dbt_marketo/pull/57) includes the following updates:
+
+## Under the Hood
+- Migrates the `union_connections`, `apply_source_relation`, and `partition_by_source_relation` macros to the `dbt_fivetran_utils` package.
+- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_marketo/#source-casing-for-case-sensitive-destinations) section of the README for details.
+
 # dbt_marketo v1.3.0
 
 [PR #54](https://github.com/fivetran/dbt_marketo/pull/54) includes the following updates:
@@ -283,4 +291,4 @@ This is a **BREAKING CHANGE** as you will need to run a full refresh due to the
 - Redshift recently included pivot as a reserved word within the warehouse. As such, the `pivot` CTE within the `marketo__change_data_pivot` and `marketo__change_data_details` models have been changed to `pivots` to avoid the Redshift error.
 
 # dbt_marketo_source v0.1.0 -> v0.5.0
-Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
+Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!

README.md

@@ -66,7 +66,7 @@ Include the following Marketo package version in your `packages.yml` file.
 ```yml
 packages:
   - package: fivetran/marketo
-    version: [">=1.3.0", "<1.4.0"]
+    version: [">=1.4.0", "<1.5.0"]
 ```
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/marketo_source` in your `packages.yml` since this package has been deprecated.
 
@@ -79,14 +79,12 @@ dispatch:
 ```
 
 ### Define database and schema variables
-
 #### Option A: Single connection
-By default, this package runs using your [destination](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile) and the `marketo` schema. If this is not where your Marketo data is (for example, if your Marketo schema is named `marketo_fivetran`), add the following configuration to your root `dbt_project.yml` file:
+By default, this package runs using your destination and the `marketo` schema. If this is not where your Marketo data is (for example, if your Marketo schema is named `marketo_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 vars:
-  marketo:
-    marketo_database: your_database_name
+    marketo_database: your_destination_name
     marketo_schema: your_schema_name
 ```
 
@@ -110,42 +108,9 @@ vars:
         name: connection_2_source_name
 ```
 
-##### Recommended: Incorporate unioned sources into DAG
-> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your Marketo connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each Marketo source rather than one set of unioned models.*
-
-By default, this package defines one single-connection source, called `marketo`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your Marketo sources, though the package will run successfully.
+#### Optional: Incorporate unioned sources into DAG
 
-To properly incorporate all of your Marketo connections into your project's DAG:
-1. Define each of your sources in a `.yml` file in the models directory of your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_marketo.yml` [file](https://github.com/fivetran/dbt_marketo/blob/main/models/staging/src_marketo.yml).
-
-```yml
-# a .yml file in your root project
-
-version: 2
-
-sources:
-  - name: <name> # ex: Should match name in marketo_sources
-    schema: <schema_name>
-    database: <database_name>
-    loader: fivetran
-    config:
-      loaded_at_field: _fivetran_synced
-      freshness: # feel free to adjust to your liking
-        warn_after: {count: 72, period: hour}
-        error_after: {count: 168, period: hour}
-
-    tables: # copy and paste from marketo/models/staging/src_marketo.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
-```
-
-> **Note**: If there are source tables you do not have (see [Enabling/Disabling Models](https://github.com/fivetran/dbt_marketo?tab=readme-ov-file#enablingdisabling-models)), you may still include them, as long as you have set the right variables to `False`.
-
-2. Set the `has_defined_sources` variable (scoped to the `marketo` package) to `True`, like such:
-```yml
-# dbt_project.yml
-vars:
-  marketo:
-    has_defined_sources: true
-```
+If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Marketo connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_marketo/blob/main/models/staging/src_marketo.yml). Set the variable `has_defined_sources: true` under the Marketo namespace in your `dbt_project.yml`. Otherwise, your Marketo connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.
 
 ### Enabling/Disabling Models
 This package takes into consideration tables that may not be synced due to slowness caused by the Marketo API. By default the `campaign`, `program`, and `activity_delete_lead` tables are enabled. If you do not sync these tables, disable the related models or fields by adding the following to your `dbt_project.yml` file:
@@ -203,6 +168,14 @@ vars:
     marketo_<default_source_table_name>_identifier: "your_table_name"
 ```
 
+#### Source casing for case-sensitive destinations
+By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:
+
+```yml
+vars:
+    fivetran_using_source_casing: true
+```
+
 #### Changing the Lead Date Range
 Because of the typical volume of lead data, you may want to limit this package's models to work with a recent date range of your Marketo data (however, note that all final models are materialized as incremental tables).
 

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'marketo'
-version: '1.3.0'
+version: '1.4.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 on-run-start: "{{ lead_history_columns_warning() }}"

integration_tests/ci/test_scenarios.yml

@@ -24,3 +24,9 @@ test_scenarios:
       marketo__enable_programs: false
       marketo__activity_delete_lead_enabled: false
     include_incremental: false
+
+  - name: "MDLS"
+    vars:
+      fivetran_using_source_casing: true
+    include_incremental: false
+    include_dbt_seed: true

integration_tests/dbt_project.yml

@@ -1,13 +1,13 @@
 name: 'marketo_integration_tests'
-version: '1.3.0'
+version: '1.4.0'
 profile: 'integration_tests'
 config-version: 2
 
 models:
   +persist_docs:
     relation: true
     columns: "{{ false if target.type in ('databricks') else true }}"
-    +schema: "marketo_{{ var('directed_schema','dev') }}"  
+  +schema: "marketo_{{ var('directed_schema','dev') }}"
 
 vars:
     marketo_schema: marketo_integration_tests
@@ -37,6 +37,7 @@ dispatch:
 seeds:
     +docs:
         show: False
+    +quote_columns: "{{ target.type == 'redshift' or var('fivetran_using_source_casing', false) }}"
     marketo_integration_tests:
         marketo_email_template_history_data:
             +column_types:
@@ -54,4 +55,4 @@ seeds:
 
 flags:
     send_anonymous_usage_stats: False
-    use_colors: True
+    use_colors: True