Datatype numeric/float update

.gitignore

@@ -4,6 +4,7 @@ package-lock.yml
 .dbt/
 dbt_modules/
 dbt_packages/
+dbt_internal_packages/
 logs/
 profiles.yml
 target/

CHANGELOG.md

@@ -1,3 +1,41 @@
+# dbt_recurly v1.4.0
+
+[PR #45](https://github.com/fivetran/dbt_recurly/pull/45) includes the following updates:
+
+## Schema/Data Change
+**22 total changes • 21 possible breaking changes**
+
+| Data Model(s) | Change type | Old | New | Fields affected |
+| ---------- | ----------- | -------- | -------- | ----- |
+| `recurly__account_overview` | Data type | `float` | `numeric` | `total_charges`, `total_credits`, `total_balance`, `total_discounts`, `charges_this_month`, `credits_this_month`, `balance_this_month`, `discounts_this_month` |
+| `recurly__account_overview` | Data type | `int` | `numeric` | `total_taxes`, `taxes_this_month` |
+| `recurly__account_daily_overview` | Data type | `float` | `numeric` | `daily_net_change`, `daily_charges`, `daily_credits`, `daily_discounts`, `rolling_account_balance`, `rolling_charge_balance`, `rolling_credit_balance`, `rolling_discount_balance` |
+| `recurly__account_daily_overview` | Data type | `int` | `numeric` | `daily_taxes`, `rolling_tax_balance` |
+| `recurly__balance_transactions` | Data type | `float` | `numeric` | `amount`, `credit_applied`, `discount`, `unit_amount` |
+| `recurly__balance_transactions` | Data type | `int` | `numeric` | `tax`, `subtotal` |
+| `recurly__monthly_recurring_revenue` | Data type | `float` | `numeric` | `current_month_mrr`, `previous_month_mrr` |
+| `recurly__subscription_overview` | Data type | `float` | `numeric` | `unit_amount` |
+| `recurly__subscription_overview` | Data type | `int` | `numeric` | `add_ons_total`, `subtotal` |
+| `recurly__line_item_enhanced` | Data type | `float` | `numeric` | `unit_amount`, `discount_amount`, `total_amount` |
+| `recurly__line_item_enhanced` | Data type | `int` | `numeric` | `tax_amount`, `refund_amount` |
+| `recurly__line_item_enhanced` | Logic update | `line_item_index` ordered by `created_at` only | `line_item_index` ordered by `created_at`, then `line_item_id` | Ensures deterministic ordering when multiple line items share the same `created_at` timestamp |
+| `int_recurly__transactions_grouped` | Data type | `float` | `numeric` | `daily_charges`, `daily_credits`, `daily_balance`, `daily_discounts` |
+| `int_recurly__transactions_grouped` | Data type | `int` | `numeric` | `daily_taxes` |
+| `int_recurly__account_cumulatives` | Data type | `float` | `numeric` | `total_charges`, `total_credits`, `total_balance`, `total_discounts`, `charges_this_month`, `credits_this_month`, `balance_this_month`, `discounts_this_month` |
+| `int_recurly__account_cumulatives` | Data type | `int` | `numeric` | `total_taxes`, `taxes_this_month` |
+| `int_recurly__account_rolling_totals` | Data type | `float` | `numeric` | `daily_charges`, `daily_credits`, `daily_balance`, `daily_discounts`, `rolling_account_balance`, `rolling_charge_balance`, `rolling_credit_balance`, `rolling_discount_balance` |
+| `int_recurly__account_rolling_totals` | Data type | `int` | `numeric` | `daily_taxes`, `rolling_tax_balance` |
+| `int_recurly__account_running_totals` | Data type | `float` | `numeric` | `daily_charges`, `daily_credits`, `daily_balance`, `daily_discounts`, `rolling_account_balance`, `rolling_charge_balance`, `rolling_credit_balance`, `rolling_discount_balance` |
+| `int_recurly__account_running_totals` | Data type | `int` | `numeric` | `daily_taxes`, `rolling_tax_balance` |
+| All `stg_recurly__*` models | Data type | `float` | `numeric` | `amount`, `balance`, `credit_applied`, `discount`, `setup_fees`, `unit_amount` |
+| All `stg_recurly__*` models | Data type | `int` | `numeric` | `add_ons_total`, `discount`, `paid`, `refundable_amount`, `subtotal`, `tax`, `total` |
+
+## Under the Hood
+- Corrected `get_*_columns` macro fields for `invoice_history`, `line_item_history`, `subscription_history` that were cast as `int` instead of `float` to match the Recurly API type (`number <float>`).
+
+## Documentation
+- Fixes link to `streamlit_fivetran_billing_model` [repo](https://github.com/fivetran/streamlit_fivetran_billing_model) in the README.
+
 # dbt_recurly v1.3.0
 
 [PR #42](https://github.com/fivetran/dbt_recurly/pull/42) includes the following updates:

README.md

@@ -46,7 +46,7 @@ By default, this package materializes the following final tables:
 ---
 
 ## Visualizations
-Many of the above reports are now configurable for [visualization via Streamlit](https://github.com/fivetran/streamlit_recurly). Check out some [sample reports here](https://fivetran-billing-model.streamlit.app/).
+Many of the above reports are now configurable for [visualization via Streamlit](https://github.com/fivetran/streamlit_fivetran_billing_model). Check out some [sample reports here](https://fivetran-billing-model.streamlit.app/).
 
 <p align="center">
 <a href="https://fivetran-billing-model.streamlit.app/">
@@ -74,7 +74,7 @@ Include the following recurly package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/recurly
-    version: [">=1.3.0", "<1.4.0"]
+    version: [">=1.4.0", "<1.5.0"]
 ```
 
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/recurly_source` in your `packages.yml` since this package has been deprecated.

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'recurly'
-version: '1.3.0'
+version: '1.4.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 

integration_tests/ci/sample.profiles.yml

@@ -12,13 +12,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: recurly_integrations_tests_1
+      schema: recurly_integrations_tests
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: recurly_integrations_tests_1
+      schema: recurly_integrations_tests
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -29,7 +29,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: recurly_integrations_tests_1
+      schema: recurly_integrations_tests
       threads: 8
     postgres:
       type: postgres
@@ -38,13 +38,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: recurly_integrations_tests_1
+      schema: recurly_integrations_tests
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: recurly_integrations_tests_1
+      schema: recurly_integrations_tests
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,10 +1,10 @@
 name: 'recurly_integration_tests'
-version: '1.3.0'
+version: '1.4.0'
 profile: 'integration_tests'
 config-version: 2
 
 vars:
-  recurly_schema: recurly_integrations_tests_1
+  recurly_schema: recurly_integrations_tests
   recurly:
     account_balance_history_identifier: "account_balance_history_data"
     account_history_identifier: "account_history_data"

macros/staging/get_invoice_history_columns.sql

@@ -18,12 +18,12 @@
     {"name": "net_terms", "datatype": dbt.type_string()},
     {"name": "currency", "datatype": dbt.type_string()},
     {"name": "balance", "datatype": dbt.type_float()},
-    {"name": "paid", "datatype": dbt.type_int()},
-    {"name": "total", "datatype": dbt.type_int()},
-    {"name": "subtotal", "datatype": dbt.type_int()},
-    {"name": "refundable_amount", "datatype": dbt.type_int()},
-    {"name": "discount", "datatype": dbt.type_int()},
-    {"name": "tax", "datatype": dbt.type_int()},
+    {"name": "paid", "datatype": dbt.type_float()},
+    {"name": "total", "datatype": dbt.type_float()},
+    {"name": "subtotal", "datatype": dbt.type_float()},
+    {"name": "refundable_amount", "datatype": dbt.type_float()},
+    {"name": "discount", "datatype": dbt.type_float()},
+    {"name": "tax", "datatype": dbt.type_float()},
     {"name": "tax_type", "datatype": dbt.type_string()},
     {"name": "tax_region", "datatype": dbt.type_string()},
     {"name": "tax_rate", "datatype": dbt.type_float()}

macros/staging/get_line_item_history_columns.sql

@@ -24,10 +24,10 @@
     {"name": "amount", "datatype": dbt.type_float()},
     {"name": "description", "datatype": dbt.type_string()},
     {"name": "quantity", "datatype": dbt.type_int()},
-    {"name": "unit_amount", "datatype": dbt.type_int()},
-    {"name": "subtotal", "datatype": dbt.type_int()},
+    {"name": "unit_amount", "datatype": dbt.type_float()},
+    {"name": "subtotal", "datatype": dbt.type_float()},
     {"name": "discount", "datatype": dbt.type_float()},
-    {"name": "tax", "datatype": dbt.type_int()},
+    {"name": "tax", "datatype": dbt.type_float()},
     {"name": "taxable", "datatype": "boolean"},
     {"name": "tax_exempt", "datatype": "boolean"},
     {"name": "tax_code", "datatype": dbt.type_string()},

macros/staging/get_subscription_history_columns.sql

@@ -26,10 +26,10 @@
     {"name": "paused_at", "datatype": dbt.type_timestamp()},
     {"name": "remaining_pause_cycles", "datatype": dbt.type_int()},
     {"name": "currency", "datatype": dbt.type_string()},
-    {"name": "unit_amount", "datatype": dbt.type_int()},
+    {"name": "unit_amount", "datatype": dbt.type_float()},
     {"name": "quantity", "datatype": dbt.type_int()},
-    {"name": "add_ons_total", "datatype": dbt.type_int()},
-    {"name": "subtotal", "datatype": dbt.type_int()},
+    {"name": "add_ons_total", "datatype": dbt.type_float()},
+    {"name": "subtotal", "datatype": dbt.type_float()},
     {"name": "collection_method", "datatype": dbt.type_string()},
     {"name": "expiration_reason", "datatype": dbt.type_string()}
 ] %}

models/staging/stg_recurly__account_balance_history.sql

@@ -23,7 +23,7 @@ final as (
         source_relation,
         account_id,
         cast(account_updated_at as {{ dbt.type_timestamp() }}) as account_updated_at,
-        cast(amount as {{ dbt.type_float() }}) as amount,
+        cast(amount as {{ dbt.type_numeric() }}) as amount,
         currency,
         past_due,
         row_number() over (partition by account_id {{ recurly.partition_by_source_relation() }} order by account_updated_at desc) = 1 as is_most_recent_record

models/staging/stg_recurly__coupon_discount.sql

@@ -22,7 +22,7 @@ final as (
     select
         source_relation,
         coupon_id,
-        cast(amount as {{ dbt.type_float() }}) as amount,
+        cast(amount as {{ dbt.type_numeric() }}) as amount,
         currency,
         fivetran_id,
         percentage,

models/staging/stg_recurly__credit_payment_history.sql

@@ -27,7 +27,7 @@ final as (
         cast(updated_at as {{ dbt.type_timestamp() }}) as updated_at,
         account_id,
         action,
-        cast(amount as {{ dbt.type_float() }}) as amount,
+        cast(amount as {{ dbt.type_numeric() }}) as amount,
         applied_to_invoice_id,
         cast(created_at as {{ dbt.type_timestamp() }}) as created_at,
         currency,

models/staging/stg_recurly__invoice_history.sql

@@ -24,28 +24,28 @@ final as (
         id as invoice_id,
         cast(updated_at as {{ dbt.type_timestamp() }}) as updated_at,
         account_id,
-        cast(balance as {{ dbt.type_float() }}) as balance,
+        cast(balance as {{ dbt.type_numeric() }}) as balance,
         cast(closed_at as {{ dbt.type_timestamp() }}) as closed_at,
         collection_method,
         cast(created_at as {{ dbt.type_timestamp() }}) as created_at,
         currency,
-        discount,
+        cast(discount as {{ dbt.type_numeric() }}) as discount,
         cast(due_at as {{ dbt.type_timestamp() }}) as due_at,
         row_number() over (partition by id {{ recurly.partition_by_source_relation() }} order by updated_at desc) = 1 as is_most_recent_record,
         net_terms,
         number,
         origin,
-        paid,
+        cast(paid as {{ dbt.type_numeric() }}) as paid,
         po_number,
         previous_invoice_id,
-        refundable_amount,
+        cast(refundable_amount as {{ dbt.type_numeric() }}) as refundable_amount,
         state,
-        subtotal,
-        tax,
+        cast(subtotal as {{ dbt.type_numeric() }}) as subtotal,
+        cast(tax as {{ dbt.type_numeric() }}) as tax,
         tax_rate,
         tax_region,
         tax_type,
-        total,
+        cast(total as {{ dbt.type_numeric() }}) as total,
         type
     from fields
 )

models/staging/stg_recurly__line_item_history.sql

@@ -26,12 +26,12 @@ final as (
         account_id,
         add_on_code,
         add_on_id,
-        cast(amount as {{ dbt.type_float() }}) as amount,
+        cast(amount as {{ dbt.type_numeric() }}) as amount,
         cast(created_at as {{ dbt.type_timestamp() }}) as created_at,
-        credit_applied,
+        cast(credit_applied as {{ dbt.type_numeric() }}) as credit_applied,
         currency,
         description,
-        discount,
+        cast(discount as {{ dbt.type_numeric() }}) as discount,
         cast(end_date as {{ dbt.type_timestamp() }}) as ended_at,
         refund as has_refund,
         invoice_id,
@@ -50,15 +50,15 @@ final as (
         cast(start_date as {{ dbt.type_timestamp() }}) as started_at,
         state,
         subscription_id,
-        subtotal,
-        tax,
+        cast(subtotal as {{ dbt.type_numeric() }}) as subtotal,
+        cast(tax as {{ dbt.type_numeric() }}) as tax,
         tax_code,
         tax_exempt,
         tax_region,
         tax_rate,
         tax_type,
         type,
-        cast(unit_amount as {{ dbt.type_float() }}) as unit_amount,
+        cast(unit_amount as {{ dbt.type_numeric() }}) as unit_amount,
         uuid
     from fields
 )

models/staging/stg_recurly__plan_currency_history.sql

@@ -24,8 +24,8 @@ final as (
         plan_id,
         cast(plan_updated_at as {{ dbt.type_timestamp() }}) as plan_updated_at,
         currency,
-        setup_fees,
-        cast(unit_amount as {{ dbt.type_float() }}) as unit_amount,
+        cast(setup_fees as {{ dbt.type_numeric() }}) as setup_fees,
+        cast(unit_amount as {{ dbt.type_numeric() }}) as unit_amount,
         row_number() over (partition by plan_id {{ recurly.partition_by_source_relation() }} order by plan_updated_at desc) = 1 as is_most_recent_record
     from fields
 )

models/staging/stg_recurly__subscription_add_on_history.sql

@@ -31,7 +31,7 @@ final as (
         plan_add_on_id,
         quantity,
         subscription_id,
-        cast(unit_amount as {{ dbt.type_float() }}) as unit_amount,
+        cast(unit_amount as {{ dbt.type_numeric() }}) as unit_amount,
         row_number() over (partition by id {{ recurly.partition_by_source_relation() }} order by updated_at desc) = 1 as is_most_recent_record
     from fields
 )

models/staging/stg_recurly__subscription_change_history.sql

@@ -33,7 +33,7 @@ final as (
         plan_id,
         quantity,
         subscription_id,
-        unit_amount,
+        cast(unit_amount as {{ dbt.type_numeric() }}) as unit_amount,
         row_number() over (partition by id {{ recurly.partition_by_source_relation() }} order by updated_at desc) = 1 as is_most_recent_record
     from fields
 ) 

models/staging/stg_recurly__subscription_history.sql

@@ -25,7 +25,7 @@ final as (
         cast(updated_at as {{ dbt.type_timestamp() }}) as updated_at,
         account_id,
         cast(activated_at as {{ dbt.type_timestamp() }}) as activated_at,
-        add_ons_total,
+        cast(add_ons_total as {{ dbt.type_numeric() }}) as add_ons_total,
         cast(canceled_at as {{ dbt.type_timestamp() }}) as canceled_at,
         collection_method,
         cast(created_at as {{ dbt.type_timestamp() }}) as created_at,
@@ -46,11 +46,11 @@ final as (
         remaining_pause_cycles,
         renewal_billing_cycles,
         state,
-        subtotal,
+        cast(subtotal as {{ dbt.type_numeric() }}) as subtotal,
         total_billing_cycles,
         cast(trial_ends_at as {{ dbt.type_timestamp() }}) as trial_ends_at,
         cast(trial_started_at as {{ dbt.type_timestamp() }}) as trial_started_at,
-        cast(unit_amount as {{ dbt.type_float() }}) as unit_amount,
+        cast(unit_amount as {{ dbt.type_numeric() }}) as unit_amount,
         uuid
 
         --The below macro adds the fields defined within your accounts_pass_through_columns variable into the staging model

models/staging/stg_recurly__transaction.sql

@@ -24,7 +24,7 @@ final as (
         id as transaction_id,
         cast(created_at as {{ dbt.type_timestamp() }}) as created_at,
         account_id,
-        cast(amount as {{ dbt.type_float() }}) as amount,
+        cast(amount as {{ dbt.type_numeric() }}) as amount,
         billing_city,
         billing_country,
         billing_first_name,

models/standardized_models/recurly__line_item_enhanced.sql

@@ -58,7 +58,7 @@ enhanced as (
         line_items.source_relation,
         line_items.invoice_id as header_id,
         line_items.line_item_id,
-        cast(row_number() over (partition by line_items.invoice_id {{ recurly.partition_by_source_relation(alias='line_items') }} order by line_items.created_at) as {{ dbt.type_int() }}) as line_item_index,
+        cast(row_number() over (partition by line_items.invoice_id {{ recurly.partition_by_source_relation(alias='line_items') }} order by line_items.created_at, line_items.line_item_id) as {{ dbt.type_int() }}) as line_item_index,
         line_items.created_at,
         line_items.currency,
         line_items.state as line_item_status,
@@ -141,7 +141,7 @@ final as (
         payment_method,
         payment_at,
         fee_amount,
-        cast(null as {{ dbt.type_float() }}) as refund_amount,
+        cast(null as {{ dbt.type_numeric() }}) as refund_amount,
         subscription_id,
         subscription_plan,
         subscription_period_started_at,
@@ -174,11 +174,11 @@ final as (
         transaction_type,
         billing_type,
         cast(null as {{ dbt.type_string() }}) as product_type,
-        cast(null as {{ dbt.type_float() }}) as quantity,
-        cast(null as {{ dbt.type_float() }}) as unit_amount,
-        cast(null as {{ dbt.type_float() }}) as discount_amount,
-        cast(null as {{ dbt.type_float() }}) as tax_amount,
-        cast(null as {{ dbt.type_float() }}) as total_amount,
+        cast(null as {{ dbt.type_int() }}) as quantity,
+        cast(null as {{ dbt.type_numeric() }}) as unit_amount,
+        cast(null as {{ dbt.type_numeric() }}) as discount_amount,
+        cast(null as {{ dbt.type_numeric() }}) as tax_amount,
+        cast(null as {{ dbt.type_numeric() }}) as total_amount,
         payment_id,
         cast(null as {{ dbt.type_string() }}) as payment_method_id,
         payment_method,