Error when running terraform plan; trouble with yaml.tftpl configuration and missing dataset

[raymondlang]

Upon running:

terraform \
    -chdir="${TERRAFORM_RUN_DIR}" \
    plan

I encounter this error, I've replaced sensitive information with

│ Error: Error in function call
│ 
│   on main.tf line 114, in resource "local_file" "feature_store_configuration":
│  114:   content = templatefile("${local.source_root_dir}/config/${var.feature_store_config_env}.yaml.tftpl", {
│  115:     project_id             = var.feature_store_project_id
│  116:     project_name           = data.google_project.feature_store_project.name
│  117:     project_number         = data.google_project.feature_store_project.number
│  118:     cloud_region           = var.google_default_region
│  119:     mds_project_id         = var.data_project_id
│  120:     mds_dataset            = "${var.mds_dataset_prefix}_${local.mds_dataset_suffix}"
│  121:     website_url            = var.website_url
│  122:     pipelines_github_owner = var.pipelines_github_owner
│  123:     pipelines_github_repo  = var.pipelines_github_repo
│  124:     #    TODO: this needs to be specific to environment.
│  125:     location  = var.destination_data_location
│  126:     time_zone = var.time_zone
│  127:     pipeline_configuration = var.pipeline_configuration
│  128:     non_ecomm_events_list = var.non_ecomm_events_list
│  129:     non_ecomm_target_event = var.non_ecomm_target_event
│  130:   })
│     ├────────────────
│     │ while calling templatefile(path, vars)
│     │ data.google_project.feature_store_project.name is "My First Project"
│     │ data.google_project.feature_store_project.number is "<PROJECT NUMBER>"
│     │ local.mds_dataset_suffix is "<DATASET SUFFIX NUMBER>"
│     │ local.source_root_dir is "../.."
│     │ var.data_project_id is "<PROJECT ID>"
│     │ var.destination_data_location is "<PROJECT ID>.analytics_<DATASET SUFFIX NUMBER>"
│     │ var.feature_store_config_env is "config"
│     │ var.feature_store_project_id is "<PROJECT ID>"
│     │ var.google_default_region is "us-central1"
│     │ var.mds_dataset_prefix is "marketing_ga4_v1"
│     │ var.non_ecomm_events_list is empty list of string
│     │ var.non_ecomm_target_event is "login"
│     │ var.pipeline_configuration is map of map of object with 9 elements
│     │ var.pipelines_github_owner is "temporarily unused"
│     │ var.pipelines_github_repo is "temporarily unused"
│     │ var.time_zone is "America/New_York"
│     │ var.website_url is "<WEBSITE URL>"
│ Call to function "templatefile" failed: ../../config/config.yaml.tftpl:566,42-81: Missing map element; This map does not have an element with the key
│ "feature-creation-lead-score-propensity"., and 9 other diagnostic(s).

Should I edit the yaml.tftpl file to skip over these keys or should I be doing something else?

I am also encountering:

│ Error: projects/<MY PROJECT ID>/datasets/purchase_propensity not found
│ 
│   with module.monitoring[0].data.google_bigquery_dataset.purchase_propensity_dataset,
│   on modules/monitor/main.tf line 274, in data "google_bigquery_dataset" "purchase_propensity_dataset":
│  274: data "google_bigquery_dataset" "purchase_propensity_dataset" {

[raymondlang]

I added the missing map elements to the variable "pipeline_configuration" to the infrastructure/terraform/variables.tf file to get past the first error of missing keys in map.

    feature-creation-lead-score-propensity = {
      execution = {
        schedule = {
          state                    = "PAUSED"
        }
      }
    }
    auto_segmentation = {
      training = {
        schedule = {
          state                    = "PAUSED"
        }
      }
      prediction = {
        schedule = {
          state                    = "PAUSED"
        }
      }
    }
    propensity_clv = {
      training = {
        schedule = {
          state                    = "PAUSED"
        }
      }
      prediction = {
        schedule = {
          state                    = "PAUSED"
        }
      }
    }
    lead_score_propensity = {
      training = {
        schedule = {
          state                    = "PAUSED"
        }
      }
      prediction = {
        schedule = {
          state                    = "PAUSED"
        }
      }
    }
    clv = {
      training = {
        schedule = {
          state                    = "PAUSED"
        }
      }
      prediction = {
        schedule = {
          state                    = "PAUSED"
        }
      }
    }
  }

[raymondlang]

I'm not sure why this code in the terraform/modules/feature_store/bigquery-datasets.tf isn't creating the resource that's supposedly missing:

# This resource creates a BigQuery dataset called `purchase_propensity`.
resource "google_bigquery_dataset" "purchase_propensity" {
  dataset_id    = local.config_bigquery.dataset.purchase_propensity.name
  friendly_name = local.config_bigquery.dataset.purchase_propensity.friendly_name
  project       = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id
  description   = local.config_bigquery.dataset.purchase_propensity.description
  location      = local.config_bigquery.dataset.purchase_propensity.location
  # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. 
  # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.feature_store.max_time_travel_hours configuration.
  max_time_travel_hours = local.config_bigquery.dataset.purchase_propensity.max_time_travel_hours
  # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. 
  # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed.
  delete_contents_on_destroy = false

  labels = {
    version = "prod"
  }

  # The lifecycle block allows you to configure the lifecycle of the dataset. 
  # In this case, the ignore_changes attribute is set to all, which means that 
  # Terraform will ignore any changes to the dataset and will not attempt to update the dataset.
  lifecycle {
    ignore_changes = all
  }
}

[chmstimoteo]

Hey @raymondlang , I'm here to help understand if there is any hiccups in your deployment.

I'm going to try deploy the solution from scratch on a brand new project. Pls, let me know where you're right now.

Carlos.

[raymondlang]

Hi @chmstimoteo , I am still running into one of the errors:

│ Error: projects/<MY PROJECT ID>/datasets/purchase_propensity not found
│ 
│   with module.monitoring[0].data.google_bigquery_dataset.purchase_propensity_dataset,
│   on modules/monitor/main.tf line 274, in data "google_bigquery_dataset" "purchase_propensity_dataset":
│  274: data "google_bigquery_dataset" "purchase_propensity_dataset" {

It seems like this dataset is not being created in Google Bigquery despite this line in the terraform.tfvars file:

deploy_feature_store = true

which should've created this resource via this block in terraform/modules/feature_store/bigquery-datasets.tf :

# This resource creates a BigQuery dataset called `purchase_propensity`.
resource "google_bigquery_dataset" "purchase_propensity" {
  dataset_id    = local.config_bigquery.dataset.purchase_propensity.name
  friendly_name = local.config_bigquery.dataset.purchase_propensity.friendly_name
  project       = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id
  description   = local.config_bigquery.dataset.purchase_propensity.description
  location      = local.config_bigquery.dataset.purchase_propensity.location
  # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. 
  # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.feature_store.max_time_travel_hours configuration.
  max_time_travel_hours = local.config_bigquery.dataset.purchase_propensity.max_time_travel_hours
  # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. 
  # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed.
  delete_contents_on_destroy = false

  labels = {
    version = "prod"
  }

  # The lifecycle block allows you to configure the lifecycle of the dataset. 
  # In this case, the ignore_changes attribute is set to all, which means that 
  # Terraform will ignore any changes to the dataset and will not attempt to update the dataset.
  lifecycle {
    ignore_changes = all
  }
}

The terraform init step did run smoothly fyi:

terraform \
    -chdir="${TERRAFORM_RUN_DIR}" \
    init

[raymondlang]

Hi,

I tried again to launch a whole new project with Google Cloudshell Editor and ran into the exact same errors

╷
│ Error: Error in function call
│ 
│   on main.tf line 114, in resource "local_file" "feature_store_configuration":
│  114:   content = templatefile("${local.source_root_dir}/config/${var.feature_store_config_env}.yaml.tftpl", {
│  115:     project_id             = var.feature_store_project_id
│  116:     project_name           = data.google_project.feature_store_project.name
│  117:     project_number         = data.google_project.feature_store_project.number
│  118:     cloud_region           = var.google_default_region
│  119:     mds_project_id         = var.data_project_id
│  120:     mds_dataset            = "${var.mds_dataset_prefix}_${local.mds_dataset_suffix}"
│  121:     website_url            = var.website_url
│  122:     pipelines_github_owner = var.pipelines_github_owner
│  123:     pipelines_github_repo  = var.pipelines_github_repo
│  124:     #    TODO: this needs to be specific to environment.
│  125:     location  = var.destination_data_location
│  126:     time_zone = var.time_zone
│  127:     pipeline_configuration = var.pipeline_configuration
│  128:     non_ecomm_events_list = var.non_ecomm_events_list
│  129:     non_ecomm_target_event = var.non_ecomm_target_event
│  130:   })
│     ├────────────────
│     │ while calling templatefile(path, vars)
│     │ data.google_project.feature_store_project.name is "My First Project"
│     │ data.google_project.feature_store_project.number is "977075964570"
│     │ local.mds_dataset_suffix is "320898935"
│     │ local.source_root_dir is "../.."
│     │ var.data_project_id is "singular-server-400422"
│     │ var.destination_data_location is "US"
│     │ var.feature_store_config_env is "config"
│     │ var.feature_store_project_id is "singular-server-400422"
│     │ var.google_default_region is "us-central1"
│     │ var.mds_dataset_prefix is "marketing_ga4_v1"
│     │ var.non_ecomm_events_list is list of string with 2 elements
│     │ var.non_ecomm_target_event is "login"
│     │ var.pipeline_configuration is map of map of object with 9 elements
│     │ var.pipelines_github_owner is "temporarily unused"
│     │ var.pipelines_github_repo is "temporarily unused"
│     │ var.time_zone is "America/New_York"
│     │ var.website_url is "https://www.precisionformedicine.com/"
│ 
│ Call to function "templatefile" failed: ../../config/config.yaml.tftpl:566,42-81: Missing map element; This map does not have an element with the key
│ "feature-creation-lead-score-propensity"., and 9 other diagnostic(s).
╵
╷
│ Error: External Program Execution Failed
│ 
│   with data.external.check_ga4_property_type,
│   on main.tf line 134, in data "external" "check_ga4_property_type":
│  134:   program     = ["bash", "-c", "${local.uv_run_alias} ga4-setup --ga4_resource=check_property_type --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id}"]
│ 
│ The data source received an unexpected error while attempting to execute the program.
│ 
│ Program: /usr/bin/bash
│ Error Message: Using CPython 3.11.11
│ Creating virtual environment at: .venv
│ Installed 95 packages in 148ms
│ Traceback (most recent call last):
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 65, in error_remapped_callable
│     return callable_(*args, **kwargs)
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/grpc/_interceptor.py", line 277, in __call__
│     response, ignored_call = self._with_call(
│                              ^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/grpc/_interceptor.py", line 332, in _with_call
│     return call.result(), call
│            ^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/grpc/_channel.py", line 440, in result
│     raise self
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/grpc/_interceptor.py", line 315, in continuation
│     response, call = self._thunk(new_method).with_call(
│                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/grpc/_channel.py", line 1198, in with_call
│     return _end_unary_response_blocking(state, call, True, None)
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
│     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
│     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│ grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
│       status = StatusCode.PERMISSION_DENIED
│       details = "The caller does not have permission"
│       debug_error_string = "UNKNOWN:Error received from peer ipv4:74.125.196.95:443 {grpc_message:"The caller does not have permission", grpc_status:7,
│ created_time:"2025-01-27T20:20:29.71355058+00:00"}"
│ >
│ 
│ The above exception was the direct cause of the following exception:
│ 
│ Traceback (most recent call last):
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/bin/ga4-setup", line 10, in <module>
│     sys.exit(entry())
│              ^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/python/ga4_setup/setup.py", line 516, in entry
│     property = get_property(configuration)
│                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/python/ga4_setup/setup.py", line 115, in get_property
│     return client.get_property(
│            ^^^^^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/google/analytics/admin_v1alpha/services/analytics_admin_service/client.py", line 1844, in get_property
│     response = rpc(
│                ^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/google/api_core/gapic_v1/method.py", line 113, in __call__
│     return wrapped_func(*args, **kwargs)
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
│     return func(*args, **kwargs)
│            ^^^^^^^^^^^^^^^^^^^^^
│   File "/home/raymond_w_lang/cloudshell_open/marketing-analytics-jumpstart/.venv/lib/python3.11/site-packages/google/api_core/grpc_helpers.py", line 67, in error_remapped_callable
│     raise exceptions.from_grpc_error(exc) from exc
│ google.api_core.exceptions.PermissionDenied: 403 The caller does not have permission
│ 
│ State: exit status 1
╵
╷
│ Error: projects/singular-server-400422/datasets/purchase_propensity not found
│ 
│   with module.monitoring[0].data.google_bigquery_dataset.purchase_propensity_dataset,
│   on modules/monitor/main.tf line 276, in data "google_bigquery_dataset" "purchase_propensity_dataset":
│  276: data "google_bigquery_dataset" "purchase_propensity_dataset" {

Are there any new fixes for these?

[GaluhPutraaa]

Same i have this error too

[raymondlang]

Hey @raymondlang , I'm here to help understand if there is any hiccups in your deployment.

I'm going to try deploy the solution from scratch on a brand new project. Pls, let me know where you're right now.

Carlos.

Hi Carlos are there any new updates for the errors: we're experiencing?