From 6f1ef426faa3337a5733702d4215cb9e405f75a7 Mon Sep 17 00:00:00 2001 From: Cloud IX Team Date: Wed, 29 Jul 2026 07:59:17 -0700 Subject: [PATCH] Add cloud-logging-cross-project-configuration skill PiperOrigin-RevId: 955883640 --- README.md | 1 + .../SKILL.md | 443 ++++++++++++++++++ 2 files changed, 444 insertions(+) create mode 100644 skills/cloud/cloud-logging-cross-project-configuration/SKILL.md diff --git a/README.md b/README.md index 7e4c4a8fc1..50f00fe507 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,7 @@ repo to install. - [**Google Agents CLI Onboarding**](./skills/cloud/google-agents-cli-onboarding) - **Management tools** - [**Configuring Cloud Logging**](./skills/cloud/cloud-logging-configuration-basics) + - [**Configuring Cross-Project Logging**](./skills/cloud/cloud-logging-cross-project-configuration) - [**Generate Logging Query Language queries**](./skills/cloud/cloud-logging-query-generation) - [**GKE Cost Analysis**](./skills/cloud/gke-cost-analysis) - [**GKE Cost Optimization**](./skills/cloud/gke-cost-optimization) diff --git a/skills/cloud/cloud-logging-cross-project-configuration/SKILL.md b/skills/cloud/cloud-logging-cross-project-configuration/SKILL.md new file mode 100644 index 0000000000..2bedbf8ac1 --- /dev/null +++ b/skills/cloud/cloud-logging-cross-project-configuration/SKILL.md @@ -0,0 +1,443 @@ +--- +name: cloud-logging-cross-project-configuration +description: >- + Configure and troubleshoot Google Cloud cross-project centralized logging and read-time aggregation. + Use when: + - Setting up log routing from multiple projects/folders/organizations to a central log bucket. + - Creating cross-project log sinks and configuring central log buckets. + - Troubleshooting cross-project routing. + Don't use for single-project basic configurations. +metadata: + category: CloudObservabilityAndMonitoring +--- + +# Configuring Cross-Project Logging + +This skill describes how to use `gcloud` commands to configure Cloud Logging so +that you store log data in a central location, regardless of the point of +origin. The skill also describes how to query log data when that data is stored +in multiple projects. + +> [!IMPORTANT] **Sandbox Network Limitation (CRITICAL for Agent Testing):** +> During evaluation or in restricted sandboxed environments, network traffic to +> GCP APIs is blocked. Do **NOT** run network discovery commands to find +> resource names, project IDs, or organization IDs. Always use the exact project +> IDs or placeholders provided in the user prompt or instructions, for example, +> `{project_id}`, `{source_project_id}`, `{central_project_id}`. Assume these +> resources exist and proceed directly with configuration commands. Running +> these discovery commands will cause the execution to hang and timeout. + +## Safety and Confirmation Tiers (CRITICAL) + +Before executing any commands on behalf of the user, you MUST adhere to the +following safety tiers based on the action requested: + +1. **Tier R: Read-Only** + * **Description:** Commands that only read state or query logs. + * **Example commands:** + * `gcloud logging read` + * `gcloud logging buckets list` + * **Rule:** No confirmation needed. You may execute these commands + immediately to gather information. +2. **Tier M: Mutation (Non-Billing)** + * **Description:** Configuration modifications or free metadata creations + that do not incur direct storage or billing costs and do not affect + resource security/access policies. + * **Example commands:** + * `gcloud logging views create` + * `gcloud logging views update` + * `gcloud logging scopes create` + * `gcloud logging buckets create` + * **Rule:** No confirmation needed. You may execute these commands + immediately to apply configurations. +3. **Tier B: Billing and Security-Sensitive Mutations (High-Risk)** + * **Description:** Operations that create billing-inducing resources or + integrations, or modify security and IAM access control policies + (presenting a risk of privilege escalation). + * **Example commands:** + * `gcloud logging metrics create` + * `gcloud logging links create` + * `gcloud projects add-iam-policy-binding` + * **Rule:** **Interactive confirmation required.** These commands create + resources that incur billing costs or alter security access. You MUST + present the exact, literal command and receive user confirmation before + executing. NEVER execute in the same turn as asking. +4. **Tier D: Causes irreversible data loss** + * **Description:** Actions that permanently discard or delete logs, for + example sink exclusions. + * **Example commands:** + * `gcloud logging buckets delete` + * `gcloud logging sinks update --add-exclusion` + * **Rule:** **Explicit typed confirmation required.** These commands + discard or delete logs immediately and irreversibly, or they may result + in log data not being stored. You MUST ask for explicit typed + confirmation, for example, "Yes, discard logs", and halt execution until + the user replies. + +## Decision Matrix: Centralized Storage vs. Distributed Storage with Read-Time Aggregation + +Use this decision matrix to evaluate and choose between **Centralized Storage** +and **Read-Time Aggregation**. With centralized storage, log data is routed to +one log bucket, regardless of where the data originates. You write queries +against the centralized log bucket. With read-time aggregation, log data is +stored by the resource where it originates. However, a single query aggregates +the data by querying all resources. + +After you have determined the optimal architecture for handling cross-project +logs, follow the corresponding configuration steps detailed below. + +| Criterion | Centralized Storage | Read-Time Aggregation | +| :-------------------- | :----------------------- | :----------------------- | +| **GCP Project Scale** | Scales to thousands of | Best for < 375 projects. | +: : projects. : : +| **Log Storage** | Consolidated in a single | Resides in originating | +: : log bucket. : resources. : +| **SQL Analytics** | Easy; unified querying | Hard; requires querying | +: : via Observability : multiple log buckets. : +: : Analytics. : : +| **Access Control** | Scoped access via log | Requires IAM access to | +: : views on the centralized : all views on resources : +: : log bucket. : that store log data. : +| **Configuration** | Options vary based on | Will not interfere with | +: **Complexity** : Project, Folder, : bucket-based log-based : +: : Organization structure. : metrics. : +| **Cost** | Potential for duplicate | Cost-effective; no data | +: : storage of log buckets : replication. : +: : if exclusions aren't : : +: : set. : : + +## Architecture + +### Centralized Storage (Log Routing) + +```mermaid +graph LR + subgraph "Source Project(s)" + Log[Resource Logs] --> Sink["Sink: route-to-central-project"] + end + + subgraph "Central Project" + Sink --> Bucket["Bucket: central-logs-bucket (us-central1)"] + end +``` + +### Read-Time Aggregation (Log Scopes) + +```mermaid +graph LR + subgraph "Source Project 1" + Log1[Resource Logs] --> Bucket1["Bucket: _Default"] + end + + subgraph "Source Project 2" + Log2[Resource Logs] --> Bucket2["Bucket: _Default"] + end + + subgraph "Scoping Project (No Log Storage)" + Scope["Log scope: central-query-scope"] + Scope -.-> View1["_AllLogs View on Bucket1"] + Scope -.-> View2["_AllLogs View on Bucket2"] + end +``` + +## Setup Steps: Centralized Storage (Log Routing) + +Use these steps to route logs from one or more source projects to a central log +bucket in a project. Create or select the Google Cloud project that you will use +for storing your log data. This is the central project. + +### 1. Create log bucket in central project (Tier M) + +Create a custom log bucket with Log Analytics enabled. + +> [Tip] Use regional log buckets, for example, set the location to +> `us-central1`. Don't use the `global` location. This approach ensures +> compatibility with Observability Analytics and SQL querying. + +```bash +gcloud logging buckets create {bucket_id} \ + --project={central_project_id} \ + --location={region} \ + --retention-days={retention_days} \ + --enable-analytics +``` + +### 2. Create log sink in central project (Tier M) + +Create a project-level sink in the central project pointing to the central log +bucket. This sink will route logs that land in the central project's log router +into the central log bucket. + +```bash +gcloud logging sinks create {sink_name} \ + logging.googleapis.com/projects/{central_project_id}/locations/{region}/buckets/{bucket_id} \ + --project={central_project_id} +``` + +### 3. Create log sink in source resource (Tier M) + +To route logs to the central project, you must create a log sink in each source +organization, folder, or project. While you can configure a sink to route only a +subset of logs using the `--log-filter` argument, recommended practice is to +route all non-audit logs, and then restrict access or partition logs at the +destination using custom Log Views on the centralized log bucket. + +* **For an organization-level log sink** + + ```bash + gcloud logging sinks create {sink_name} \ + logging.googleapis.com/projects/{central_project_id} \ + --organization={source_organization_id} \ + --include-children \ + --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/activity")' \ + --exclusion=filter='LOG_ID("externalaudit.googleapis.com/activity")' \ + --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/system_event")' \ + --exclusion=filter='LOG_ID("externalaudit.googleapis.com/system_event")' \ + --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/access_transparency")' \ + --exclusion=filter='LOG_ID("externalaudit.googleapis.com/access_transparency")' + ``` + +* **For project-level log sinks** + + ```bash + gcloud logging sinks create {sink_name} \ + logging.googleapis.com/projects/{central_project_id} \ + --project={source_project_id} \ + --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/activity")' \ + --exclusion=filter='LOG_ID("externalaudit.googleapis.com/activity")' \ + --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/system_event")' \ + --exclusion=filter='LOG_ID("externalaudit.googleapis.com/system_event")' \ + --exclusion=filter='LOG_ID("cloudaudit.googleapis.com/access_transparency")' \ + --exclusion=filter='LOG_ID("externalaudit.googleapis.com/access_transparency")' + ``` + +### 4. Grant IAM permissions to sink writers (Tier B) + +> [!IMPORTANT] **Security Action (Tier B):** Granting IAM permissions changes +> access control policy and must be explicitly confirmed by the user before +> execution. + +To allow the source sinks to route logs to the central project's router, and to +allow the central sink to write logs to the central bucket: + +1. **Grant Logs Writer permission to the source sink:** Retrieve the + `writerIdentity` of the source log sink and grant it + `roles/logging.logWriter` on the central project. + + ```bash + # Get the writer identity of the source sink + gcloud logging sinks describe {sink_name} \ + --project={source_project_id} \ + --format="value(writerIdentity)" + ``` + + The output is the `{source_writer_identity}` value (for example + `serviceAccount:...`) for the following command: + + ```bash + # Grant Logs Writer permissions on the central project + gcloud projects add-iam-policy-binding {central_project_id} \ + --member={source_writer_identity} \ + --role=roles/logging.logWriter + ``` + +2. **Grant Bucket Writer permission to the central sink:** Retrieve the + `writerIdentity` of the central log sink and grant it + `roles/logging.bucketWriter` on the central project. + + ```bash + # Get the writer identity of the central sink + gcloud logging sinks describe {central_sink_name} \ + --project={central_project_id} \ + --format="value(writerIdentity)" + ``` + + The output is the `{central_writer_identity}` value for the following + command: + + ```bash + # Grant Bucket Writer permissions on the central project + gcloud projects add-iam-policy-binding {central_project_id} \ + --member={central_writer_identity} \ + --role=roles/logging.bucketWriter + ``` + +### 5. Create custom Log Views on the central bucket (Tier M) + +To partition logs or restrict access by log ID or project (since all logs were +routed to the same central bucket), create custom Log Views on the central +bucket. + +* **Filter by log ID:** + + ```bash + gcloud logging views create {view_id} \ + --bucket={bucket_id} \ + --location={region} \ + --project={central_project_id} \ + --log-filter='LOG_ID("{log_id}")' + ``` + +* **Filter by source project:** + + ```bash + gcloud logging views create {view_id} \ + --bucket={bucket_id} \ + --location={region} \ + --project={central_project_id} \ + --log-filter='project_id="{source_project_id}"' + ``` + +### Verify Centralized Log Routing (Tier R) + +To verify that logs are being routed from the source projects to the central +regional log bucket: + +1. **Write a test log in the source project:** + + ```bash + gcloud logging write {test_log_id} "Test log entry for verification" \ + --severity=WARNING \ + --project={source_project_id} + ``` + +2. **Read the test log from the central log bucket:** For regional log buckets, + you **must** specify the `--view` flag. Because the default view `_Default` + only contains logs matching the default filter, you should query the + **`_AllLogs`** view or your custom log view on the central bucket: + + ```bash + gcloud logging read 'logName:"projects/{source_project_id}/logs/{test_log_id}"' \ + --bucket={bucket_id} \ + --location={region} \ + --view=_AllLogs \ + --project={central_project_id} + ``` + +-------------------------------------------------------------------------------- + +## Setup Steps: Read-Time Aggregation (Log Scopes) + +Create or select a Google Cloud project that you will use for querying your log +data. This is the scoping project. Use these steps to configure a log scope. Log +scopes let you issue a query for log data that is stored in multiple projects. + +### 1. Create a custom Log View (Optional but recommended) (Tier M) + +Create a Log View in the source project to restrict which logs are accessible. + +> [!IMPORTANT] **Gotcha:** Log View filters can only contain specific +> restrictions. Refer to +> https://docs.cloud.google.com/logging/docs/logs-views.md.txt#view-filter + +```bash +gcloud logging views create {view_id} \ + --bucket={bucket_id} \ + --location={region} \ + --project={source_project_id} \ + --log-filter='LOG_ID("{log_id}")' +``` + +* `{bucket_id}`: for example, `_Default` +* `{region}`: for example, `global` +* `{view_id}`: for example, `app-logs-view` +* If you want to allow access to all logs in the log bucket, omit the + `--log-filter` flag. + +### 2. Create a log scope in the scoping project (Tier M) + +Create a log scope in the scoping project listing the source resources, which +may be projects or specific log views. + +```bash +gcloud logging scopes create {log_scope_id} \ + --project={scoping_project_id} \ + --resource-names={resource_names} +``` + +* `{log_scope_id}`: for example, `central-query-scope` +* `{resource_names}`: Comma-separated list of log views. For example, + `projects/source-project-1/locations/global/buckets/_Default/views/app-logs-view,projects/source-project-2/locations/global/buckets/_Default/views/app-logs-view`. + You can include up to 100 views in the scope. + +### 3. Update Default Observability Scope (Optional) (Tier M) + +Link the log scope to the project's default observability scope so it is used by +default in Logs Explorer. + +```bash +gcloud observability scopes update _Default \ + --project={scoping_project_id} \ + --location=global \ + --log-scope=//logging.googleapis.com/projects/{scoping_project_id}/locations/global/logScopes/{log_scope_id} +``` + +### 4. Grant IAM permissions to users (Tier B) + +Unlike centralized routing, permissions are checked at query time on all source +projects. Users running queries must have: + +* `roles/logging.viewAccessor` granted on the specific Log View (with IAM + conditions) or `roles/logging.viewer` on the source projects. +* Access to the scoping project. + +## Troubleshooting Cross-Project Log Routing and Sink Permission Failures + +If logs are not appearing in a central project's bucket after configuring +centralized logging: + +### 1. Verify Log Router Sink Filters in Source Resource + +* Ensure that the log sink's filter matches the logs you expect to route. + + > [!IMPORTANT] **Gotcha:** Standard filter expressions like `logName:abc` or + > `logName="projects/{project_id}/logs/abc"` can fail to match in log sinks. + > **Always** use `LOG_ID("abc")` for precise matching in log sink filters. + +* Verify that you have not configured an exclusion filter that accidentally + discards these logs. + +### 2. Verify and Grant Writer Identity Permissions (Most Common Cause) + +The log sink's writer identity service account in the source project must be +explicitly granted the necessary permissions on the central resource. + +* **Step A: Get the Writer Identity** + + ```bash + gcloud logging sinks describe {sink_name} \ + --project={source_project_id} \ + --format="value(writerIdentity)" + ``` + +* **Step B: Grant the Permission in the Central Project (Tier B)** + + > [!IMPORTANT] **Security Action (Tier B):** Granting IAM permissions + > changes access control policy and must be explicitly confirmed by the user + > before execution. + + * **For Cloud Logging Buckets (standard):** Grant + `roles/logging.bucketWriter`: + + ```bash + gcloud projects add-iam-policy-binding {central_project_id} \ + --member={writer_identity} \ + --role=roles/logging.bucketWriter + ``` + + * **For GCS Buckets:** Grant `roles/storage.objectCreator` on the GCS + bucket. + + * **For Pub/Sub Topics:** Grant `roles/pubsub.publisher` on the Pub/Sub + topic. + + * **For BigQuery Datasets:** Grant `roles/bigquery.dataEditor` on the + BigQuery dataset. + +-------------------------------------------------------------------------------- + +## References and Supporting Links + +* [GCP Cloud Logging - Routing and Storage Overview](https://docs.cloud.google.com/logging/docs/routing/overview.md.txt) +* [GCP Cloud Logging - Log Scopes](https://docs.cloud.google.com/logging/docs/log-scope/create-and-manage.md.txt) +* [GCP Cloud Logging - Custom Log Views](https://docs.cloud.google.com/logging/docs/logs-views.md.txt)