Skip to content
BranchesTags

Repository files navigation

🚀 Enterprise TIBCO BWCE Multi-Cluster GitOps Platform with Datadog Observability

Generated by Reviewed & Hardened by TIBCO BWCE Datadog APM Jenkins ArgoCD Argo Rollouts OpenShift Kubernetes JCasC & Job DSL SLSA Level 3 Sigstore Cosign CycloneDX SBOM Aqua Trivy External Secrets Operator 12-Factor App License

An enterprise-grade, multi-cluster continuous integration and progressive delivery platform purpose-built for TIBCO BusinessWorks™ Container Edition (BWCE 2.9.2 / 2.10.0) microservices on Red Hat OpenShift 4.20+. It integrates Jenkins Configuration as Code (JCasC), Dynamic Git Parameter Plugin Dropdowns, Multi-Remote SCM, ArgoCD 3.5 Multi-Cluster GitOps, and Datadog Full-Stack Observability (APM, DogStatsD, CI Visibility, Logs, and Argo Rollouts Metric Analysis).

Important

⚠️ Architectural Blueprint & Sovereign Environment Notice

  • Non-Production Testing Status: This repository serves as an architectural reference blueprint, educational design pattern, and foundational boilerplate. It has not been tested or executed in a live production environment.
  • Sovereign & Air-Gapped Boilerplate: This project is engineered as a deterministic, version-controlled Infrastructure-as-Code (IaC) template that can be safely used as a standardized boilerplate in enterprise environments where Agentic AI / LLMs are prohibited due to data sovereignty, compliance regulations, intellectual property protection, or air-gapped security policies.
  • Enterprise Hardening: Platform engineers must review, adapt, and validate all cluster endpoints, TLS certificates, HashiCorp Vault configurations, Datadog API keys, and RBAC quotas prior to production adoption.
  • AI Model Attribution & Review: Originally generated using Gemini 3.7 Flash with Antigravity, subsequently audited, verified, and hardened using Gemini 3.8 Flash.
  • Origin & Real-World Heritage: This blueprint is based on nubenetes/jenkins-git-parameter and nubenetes/jenkins-git-parameter-global-vars, drawing directly from the author's personal hands-on experience designing and operating enterprise integration platforms with these exact requirements (engineered just before agentic AI became widespread).

Tip

🔗 Companion Repository (Single Source of Truth)

This orchestration platform works in tandem with its centralized configuration repository:
🌐 nubenetes/jenkins-git-parameter-bwce-global-vars
(Houses multi-cluster environment topologies, externalized .substvar BWCE profiles, Helm values, and Vault secret references)

📑 Table of Contents

Executive Summary & Architecture Overview

This platform delivers an automated, governed, multi-cluster CI/CD and GitOps delivery pipeline for enterprise TIBCO BWCE workloads across three OpenShift clusters (DEV, STAGING, PROD).

🗺️ Click to expand: End-to-End Multi-Cluster Platform Topology Diagram 
flowchart TB
    subgraph DevSystems ["1. Developer & Portals"]
        Dev["Developer / Ops"]
        Backstage["Backstage IDP<br/>Service Catalog"]
        ITSM["Jira Service Mgmt<br/>ServiceNow CMDB"]
    end

    subgraph OCP_DEV ["Cluster 1: OCP DEV"]
        direction TB
        subgraph JenkinsPlatform ["Jenkins Control Plane"]
            Master["Jenkins Master<br/>2.492.2 LTS"]
            Seed["00-Seed-Job<br/>Orchestrator"]
            CIJob["01-CI-Build<br/>BWCE App Git"]
            CDJob["02-CD-Release<br/>Global Vars Git"]
        end

        subgraph Agents ["Agent Pods"]
            BwceAgent["bwce-builder<br/>EAR & Tests"]
            GitOpsAgent["argocd-gitops<br/>Skopeo & Cosign"]
        end

        subgraph OCPDevRegistry ["DEV Registry"]
            DevReg["Internal Registry<br/>dev-bwce"]
        end

        subgraph ArgoCDMaster ["ArgoCD 3.5"]
            ArgoServer["ArgoCD Server<br/>ApplicationSets"]
        end

        subgraph DatadogAgentStack ["Datadog Stack"]
            DDAgent["Datadog Agent<br/>APM:8126 | DSD:8125"]
            DDCluster["Cluster Agent<br/>Metrics Provider"]
        end

        DevApps["BWCE DEV Apps<br/>dev-bwce"]
    end

    subgraph OCP_STG ["Cluster 2: OCP STAGING"]
        StgReg["STG Registry"]
        StgApps["BWCE STAGING Apps<br/>staging-bwce"]
    end

    subgraph OCP_PRD ["Cluster 3: OCP PROD"]
        PrdReg["PROD Registry"]
        PrdApps["BWCE PROD Apps<br/>prod-bwce"]
        RolloutCtrl["Argo Rollouts<br/>Canary Controller"]
    end

    subgraph DatadogCloud ["Datadog Cloud"]
        DDCIVis["CI Visibility<br/>Pipeline Spans"]
        DDAPM["APM Tracing<br/>Live Profiling"]
        DDDash["Dashboards<br/>& Monitors"]
    end

    Dev -->|"Selects Branch"| CIJob
    Dev -->|"Selects Config"| CDJob
    Backstage -->|"REST API"| CDJob
    ITSM -->|"Webhook"| CDJob

    CIJob -->|"Launches"| BwceAgent
    BwceAgent -->|"Pushes Image"| DevReg
    CIJob -->|"Triggers CD"| CDJob

    CDJob -->|"Launches"| GitOpsAgent
    GitOpsAgent -->|"Skopeo Copy"| StgReg
    GitOpsAgent -->|"Skopeo Copy"| PrdReg
    GitOpsAgent -->|"Sync & Health"| ArgoServer

    ArgoServer -->|"GitOps"| DevApps
    ArgoServer -->|"GitOps"| StgApps
    ArgoServer -->|"Sync Waves"| PrdApps

    Master -.->|"CI Spans"| DDAgent
    DevApps -.->|"APM & Metrics"| DDAgent
    StgApps -.->|"APM & Metrics"| DDAgent
    PrdApps -.->|"APM & Metrics"| DDAgent
    DDAgent -->|"HTTPS"| DatadogCloud
    RolloutCtrl -.->|"Query Metrics"| DatadogCloud
Loading

📋 Architectural Breakdown & Workflow Steps:

  • Developer & External Ingestion: Developers and ITSM platforms select TIBCO BWCE source branches and global configuration tags via Jenkins UI parameters.
  • Jenkins Controller (DEV Cluster): JCasC bootstraps the Seed Job, provisioning CI build pipelines and CD release orchestrators via Job DSL.
  • Pipeline 01 (CI Build): Ephemeral bwce-builder agent compiles .ear archives via bw6-maven-plugin, executes BWUnit tests, layers the EAR onto tibco/bwce:2.9.2, and triggers downstream CD.
  • Pipeline 02 (CD Release Orchestrator): Ephemeral agent promotes BWCE images via Skopeo, updates .substvar profile bindings in jenkins-git-parameter-bwce-global-vars, and calls argoAppSync.
  • Datadog Full-Stack Observability: Datadog Cluster Agent and APM tracing (dd-java-agent.jar) provide continuous performance monitoring across DEV, STAGING, and PROD.

💡 Architectural Summary & Conclusion: This topology modernizes enterprise TIBCO BWCE deployments by standardizing container compilation, separating CI from CD release promotion, and enabling multi-cluster OpenShift delivery with automated Datadog tracing.

TIBCO BWCE Cloud-Native Best Practices

1. 12-Factor Profile Externalization via .substvar

In enterprise TIBCO BWCE implementations, building environment-specific EAR files violates 12-Factor principles. Instead:

  • Build Once: A single immutable .ear artifact (tibco-bwce-order-service_2.1.0.ear) is compiled during CI and packaged into the container image (FROM tibco/bwce:2.9.2).
  • Externalize Profiles: Configuration tokens (DEV.substvar, STAGING.substvar, PROD.substvar) reside in the Single Source of Truth configuration repository: jenkins-git-parameter-bwce-global-vars.
  • Runtime Injection: The runtime profile is selected via BW_PROFILE environment variable or ConfigMap volume mount at container startup.
⚙️ Click to expand: TIBCO BWCE Profile Externalization Flow Diagram 
flowchart LR
    subgraph BuildTime ["1. CI Packaging"]
        Source[".bwp Sources"] --> Maven["Maven Plugin"]
        Maven --> EAR["Single EAR"]
        EAR --> BaseImage["tibco/bwce:2.9.2"]
        BaseImage --> Image["Container Image<br/>(Immutable)"]
    end

    subgraph GitOpsSSOT ["2. Global Vars SSOT"]
        DevVars["DEV.substvar<br/>(dev.yaml)"]
        StgVars["STAGING.substvar<br/>(staging.yaml)"]
        PrdVars["PROD.substvar<br/>(prod.yaml)"]
    end

    subgraph RuntimeDeploy ["3. OpenShift Runtime"]
        Image --> OCPDev["DEV Pod<br/>Profile: DEV"]
        Image --> OCPStg["STG Pod<br/>Profile: STG"]
        Image --> OCPPrd["PROD Pod<br/>Profile: PROD"]

        DevVars -.->|"ConfigMap"| OCPDev
        StgVars -.->|"ConfigMap"| OCPStg
        PrdVars -.->|"ConfigMap"| OCPPrd
    end
Loading

📋 Profile Externalization Mechanics:

  • Build-Time EAR Immutability: The .ear archive contains all default process schemas, modules, and .substvar profile definitions.
  • Single Container Image: Layering the compiled EAR onto tibco/bwce:2.9.2 produces an immutable container image promoted across all environments.
  • Runtime Profile Binding: OpenShift deployment manifests inject BW_PROFILE: DEV.substvar, STAGING.substvar, or PROD.substvar at pod startup.

💡 Architectural Summary & Conclusion: Strict profile externalization achieves 12-Factor parity for TIBCO BWCE. Building the EAR once and externalizing environment variables into .substvar profiles prevents environment-specific re-compilation.

2. Engine Sizing & Performance Tuning

The BusinessWorks Container Edition runtime engine requires deliberate sizing parameters adapted to container cgroups:

Environment Replicas CPU Request (Burstable) Memory Req / Limit BW_ENGINE_THREADCOUNT BW_STEP_FLOWLIMIT BW_LOGLEVEL
DEV 2 250m (No CPU Limit) 512Mi / 1024Mi 16 50 DEBUG
STAGING 3 500m (No CPU Limit) 768Mi / 1536Mi 32 100 INFO
PROD 6 1000m (No CPU Limit) 1024Mi / 2048Mi 64 250 WARN

Tip

💡 Why CPU Limits are Omitted at the Pod Level (Preventing Linux CFS Bandwidth Throttling)

  • The CFS Quota Problem in Multi-Threaded Runtimes: Setting resources.limits.cpu instructs the Linux kernel Completely Fair Scheduler (CFS) to enforce hard bandwidth quotas (cfs_quota_us over 100ms periods). Because TIBCO BWCE and JVM garbage collection are highly concurrent (e.g. 16–64 engine threads), parallel thread execution can consume the quota in milliseconds, causing the container to be hard-throttled for the remainder of the period. This induces artificial p99 latency spikes and timeouts even when the OpenShift worker node has surplus CPU capacity.
  • Namespace-Level ResourceQuota Governance: Instead of per-pod CPU limits, capacity governance and noisy-neighbor protection are enforced at the OpenShift Project/Namespace boundary using security/openshift-namespace-resource-quota.yaml (ResourceQuota for aggregate requests.cpu, requests.memory, and limits.memory).
  • Deterministic Memory Limits: Unlike CPU (which is compressible), Memory is strictly capped with limits.memory to prevent out-of-memory cascading to neighbor pods.
  • BW_ENGINE_THREADCOUNT: Specifies the number of engine worker threads executing process instances concurrently.
  • BW_STEP_FLOWLIMIT: Prevents unbounded memory growth during traffic bursts by capping active in-memory process transitions.
  • BW_CONTAINER_SHUTDOWN_TIMEOUT_SECONDS: Set to 30s (DEV/STAGING) and 45s (PROD) for clean process draining upon SIGTERM.
  • JVM Options: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent ensuring GC predictability inside Linux container cgroups.

3. OpenShift Hardening & Health Probing

  • Security Context Constraints: Complies with OpenShift restricted-v2 SCC (non-root uid: 1001, dropped capabilities ALL, read-only root filesystems).
  • Probes:
    • Liveness Probe: HTTP GET http://:8090/health (Initial delay: 45s, Period: 15s, Timeout: 5s).
    • Readiness Probe: HTTP GET http://:8090/health (Initial delay: 20s, Period: 10s, Timeout: 3s).

Datadog Full-Stack Observability Architecture

Replaces OpenTelemetry and Grafana with an enterprise Datadog observability stack.

📊 Click to expand: Datadog Full-Stack Observability Architecture Diagram 
flowchart TB
    subgraph Sources ["1. OpenShift Workloads"]
        direction TB
        Jenkins["Jenkins Master<br/>(Datadog Plugin)"]
        BWCEApp["TIBCO BWCE App<br/>(dd-java-agent)"]
        RolloutCtrl["Argo Rollouts<br/>(Canary Engine)"]
    end

    subgraph DaemonSetLayer ["2. Node Agent Layer"]
        direction TB
        DDAgent["Datadog Agent<br/>(DaemonSet)<br/>──────────────<br/>APM Port: 8126<br/>StatsD Port: 8125<br/>Metrics: 8090<br/>Logs: JSON"]
    end

    subgraph CloudPlatform ["3. Datadog Cloud Platform"]
        direction TB
        CIVisibility["CI Visibility<br/>Pipeline Spans<br/>Queue Metrics"]
        APMTrace["APM Tracing<br/>HTTP & JMS<br/>Distributed Spans"]
        OpenMetrics["Engine Metrics<br/>Thread Pools<br/>JVM GC Pauses"]
        Dashboards["Dashboards<br/>• Jenkins CI<br/>• BWCE Engine<br/>• ArgoCD Sync"]
        Monitors["Alert Monitors<br/>• 5xx > 1.0%<br/>• p99 > 500ms<br/>• Queue Spikes"]
    end

    Jenkins -->|"CI Spans"| DDAgent
    BWCEApp -->|"APM: 8126"| DDAgent
    BWCEApp -->|"Metrics: 8090"| DDAgent
    BWCEApp -->|"Logs"| DDAgent

    DDAgent -->|"HTTPS"| CIVisibility
    DDAgent -->|"HTTPS"| APMTrace
    DDAgent -->|"HTTPS"| OpenMetrics

    CIVisibility --> Dashboards
    APMTrace --> Dashboards
    OpenMetrics --> Dashboards

    CIVisibility --> Monitors
    APMTrace --> Monitors
    OpenMetrics --> Monitors

    RolloutCtrl -.->|"Query SLA"| APMTrace
Loading

📋 Datadog Telemetry Data Flow:

  • JVM APM Instrumentation: Java agent (dd-java-agent.jar) instruments BWCE runtime processes, capturing REST endpoints, JDBC queries, and JMS activities.
  • DogStatsD Metrics: Workload pods emit custom business metrics and engine throughput statistics over UDP to the Datadog Cluster Agent.
  • Unified Dashboards: SRE teams visualize distributed traces, service dependencies, and live log streams in unified Datadog APM dashboards.

💡 Architectural Summary & Conclusion: Full-stack Datadog instrumentation provides deep visibility into complex enterprise integration flows, enabling instant bottleneck diagnosis and automated SLA monitoring.

1. Jenkins CI/CD Visibility Plugin

The official Datadog Jenkins Plugin (datadog:5.9.0) is configured via JCasC (jcasc/jenkins-jcasc.yaml):

  • Automatically correlates pipeline builds, stage execution times, agent queue wait times, and test results.
  • Injects Datadog Trace IDs (x-datadog-trace-id) across pipeline steps.
  • Global tags: env:dev, team:integration-platform, tech:tibco-bwce, cluster:ocp-dev.

2. TIBCO BWCE Java APM Tracer & DogStatsD

  • Java Tracer: The Datadog Java APM agent (dd-java-agent.jar) is injected into the container image and attached to the BWCE engine via: 
    BW_JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC -javaagent:/opt/datadog/dd-java-agent.jar -Ddd.service=tibco-bwce-order-service -Ddd.logs.injection=true -Ddd.profiling.enabled=true"
  • OpenMetrics Autodiscovery: Pod annotations configure the Datadog Agent to scrape BWCE management metrics from port 8090: 
    annotations:
  ad.datadoghq.com/tibco-bwce-order-service.logs: '[{"source": "tibco-bwce", "service": "tibco-bwce-order-service"}]'
  ad.datadoghq.com/tibco-bwce-order-service.check_names: '["openmetrics"]'
  ad.datadoghq.com/tibco-bwce-order-service.init_configs: '[{}]'
  ad.datadoghq.com/tibco-bwce-order-service.instances: |
    [{"openmetrics_endpoint": "http://%%host%%:8090/metrics", "namespace": "tibco_bwce", "metrics": [".*"]}]

3. Datadog Live Dashboards & Monitors

Preconfigured Datadog JSON dashboards and alert monitors located in observability/:

4. Progressive Delivery with Argo Rollouts & Datadog Metrics SLA

During canary deployments, Argo Rollouts queries Datadog metrics directly using sample-apps/tibco-bwce-order-service/rollout/analysis-template-datadog.yaml:

  • Error Rate SLA: Evaluates that 5xx errors $\le 0.1%$ via sum:trace.tibco_bwce_order_service.request.errors{env:prod}.as_count() / sum:trace.tibco_bwce_order_service.request.hits{env:prod}.as_count().
  • Latency SLA: Evaluates that p99 latency $\le 250 ext{ms}$ via p99:trace.tibco_bwce_order_service.request.duration{env:prod} before traffic step promotion.
🐣 Click to expand: Argo Rollouts Canary Traffic Splitting & Datadog Metric Analysis Diagram 
flowchart LR
    Ingress["OpenShift Route<br/>Ingress Traffic"]
    Canary["Canary Pods<br/>10% -> 25% -> 50%<br/>v2.1.0 (New EAR)"]
    Stable["Stable Pods<br/>90% -> 75% -> 50%<br/>v2.0.8 (Stable EAR)"]
    DatadogMetrics["Datadog APM API<br/>5xx <= 0.1%<br/>p99 <= 250ms"]
    RolloutCtrl["Argo Rollouts<br/>Controller"]

    Ingress -->|"Traffic Split"| Canary
    Ingress -->|"Traffic Split"| Stable
    Canary -.->|"APM Metrics"| DatadogMetrics
    DatadogMetrics -.->|"SLA Evaluation"| RolloutCtrl
    RolloutCtrl -->|"Pass: Promote<br/>Fail: Rollback"| Ingress
Loading

📋 Progressive Canary Delivery Steps:

  • Traffic Routing: Argo Rollouts splits OpenShift route traffic, routing 20% to canary BWCE pods and 80% to stable pods.
  • Datadog SLA Analysis: Automated AnalysisRun queries Datadog metrics (error rate $&lt; 0.5%$ and p95 latency $&lt; 200 ext{ms}$).
  • Automated Rollout / Rollback: If metrics satisfy SLAs, traffic scales to 100%; if SLAs breach, instant automated rollback redirects traffic to stable pods.

💡 Architectural Summary & Conclusion: Data-driven progressive delivery with Datadog metric analysis eliminates deployment risk for critical TIBCO integration services, ensuring zero downtime and automated fault protection.

Multi-Repository Git Parameter CI/CD Patterns

The Core Problem: Why Standard Pipelines Fail with Multiple Git Repositories

In cloud-native architectures, applications separate source code (tibco-bwce-order-service) from centralized configuration and Helm values (jenkins-git-parameter-bwce-global-vars).

When engineering teams attempt to introduce multiple gitParameter dropdowns into a single Jenkins Pipeline, they encounter a hard blocker where GitSCM appears to only support one repository, or secondary dropdowns fail.

⚠️ Click to expand: Jenkins SCM Lifecycle & Pre-Execution Blindspot Diagram 
flowchart TB
    subgraph UI_Phase ["1. Pre-Execution (Master)"]
        direction TB
        User["User opens<br/>Build UI Form"]
        Master["Master reads<br/>Job XML SCM"]
        GitParam["git-parameter<br/>queries remote refs"]
        Dropdown["Renders Dropdown<br/>for Primary Repo"]

        User --> Master --> GitParam --> Dropdown
    end

    subgraph Runtime_Phase ["2. Runtime Phase (Agent)"]
        direction TB
        AllocAgent["Ephemeral Agent<br/>Pod Allocated"]
        RunStage["Pipeline Stage:<br/>checkout(repo-2)"]

        AllocAgent --> RunStage
    end

    Gap["SCM Blindspot:<br/>Dynamic checkouts<br/>run during build and are<br/>invisible at UI render"]

    Dropdown -.-> Gap
    Gap -.-> RunStage
Loading

📋 Why Multi-Repo Pipelines Fail for BWCE:

  • Pre-Execution Render Paradox: Jenkins Master parses gitParameter dropdowns before agent allocation, discovering only repositories statically declared in Job XML.
  • Dynamic SCM Blindspot: Secondary checkouts for .substvar files executed inside pipeline stages are invisible during initial UI rendering.
  • The Pattern Solution: Pattern 1 resolves this via multi-remote refspecs in Job DSL, while Pattern 2 separates pipelines into distinct build and release phases.

💡 Architectural Summary & Conclusion: Overcoming the SCM pre-execution blindspot ensures reliable parameter rendering when coordinating BWCE source code with central global configuration repositories.

Master Seed Job & JCasC Automation Architecture

The entire CI/CD platform is self-bootstrapping and self-healing using Jenkins Configuration as Code (JCasC) combined with the Job DSL Plugin.

⚙️ Click to expand: Seed Job & Job DSL Pipeline Provisioning Flow Diagram 
flowchart TB
    subgraph JCasC_Phase ["1. Controller Bootstrap (JCasC)"]
        direction TB
        JCasC["jenkins-jcasc.yaml<br/>(Master Config)"] -->|"Registers"| SeedJob["00-Seed-Job<br/>(Platform Seed)"]
    end

    subgraph SCM_Phase ["2. SCM Synchronization"]
        direction TB
        GitRepo["Git Repository<br/>(Platform Code)"]
        SeedJob -->|"Polls H/15 * * * *"| GitRepo
        SeedJob -->|"Evaluates"| JobDSLFiles["Target Scripts<br/>(jobdsl/*.groovy)"]
    end

    subgraph Materialization ["3. Pipeline Materialization"]
        direction TB
        JobDSLFiles -->|"seed-job.groovy"| Folders["Creates Folders:<br/>• CI Pipelines<br/>• CD Orchestrators"]
        JobDSLFiles -->|"pipelines-ci.groovy"| CIPipelines["Generates CI:<br/>• order-service-ci<br/>• customer-api-ci"]
        JobDSLFiles -->|"pipelines-cd.groovy"| CDPipelines["Generates CD:<br/>• release-orchestrator<br/>• hotfix-deploy"]
    end

    Folders -.->|"Parent"| CIPipelines
    Folders -.->|"Parent"| CDPipelines
Loading

📋 Pipeline Provisioning Chain:

  • JCasC Controller Bootstrap: Jenkins initializes and configures 00-Seed-Job-Platform-Orchestrator.
  • Dynamic Polling: Polls the repository every 15 minutes for updates to Job DSL Groovy scripts.
  • Pipeline Generation: Materializes folder structures, BWCE CI pipelines (pipelines-ci.groovy), and CD orchestrators (pipelines-cd.groovy).
  • Automated Pruning: Deletes obsolete jobs automatically when removed from version control.

💡 Architectural Summary & Conclusion: Managing BWCE pipelines as version-controlled Job DSL code eliminates manual configuration drift and enables automated onboarding of enterprise integration services.

1. Why jobdsl/seed-job.groovy Only Declares Folders (Separation of Concerns)

In enterprise Jenkins-as-Code implementations, we deliberately apply the Separation of Concerns (SoC) principle:

  • Taxonomy & Hierarchy Initialization: jobdsl/seed-job.groovy acts as the root taxonomy initializer. It creates and configures the top-level folder containers (01-CI-Build-Pipelines and 02-CD-Release-Orchestrators) along with their descriptions and metadata.
  • Decoupled Pipeline Generation Logic: The application CI pipelines (jobdsl/pipelines-ci.groovy) and CD release orchestrators (jobdsl/pipelines-cd.groovy) contain iterative business logic (looping over the apps inventory, configuring multi-remote refspecs, Git Parameter dropdowns, log rotation, and Datadog CI visibility). Keeping folder declarations separate prevents monolithic scripts and enables modular additions of new business domains.

2. Managing Folders in Job DSL vs. Static JCasC

While folders can technically be defined directly in jenkins-jcasc.yaml, managing them through Job DSL is the recommended SRE practice for multi-tenant platforms:

Architectural Metric Static JCasC Management (jenkins-jcasc.yaml) Dynamic Job DSL Management (jobdsl/seed-job.groovy)
Creation Lifecycle Executed strictly upon Jenkins Controller startup or manual JCasC reload. Evaluated continuously on every Seed Job execution (automated SCM polling or manual trigger).
Orphaned Item Pruning Does not automatically purge folders or child jobs if deleted from configuration. Automated Pruning: removedJobAction('DELETE') and removedViewAction('DELETE') cleanly destroy removed jobs/folders.
Configuration Ownership Belongs to Controller System Configuration (Platform Admin scope). Belongs to Version-Controlled GitOps Code (Developer / Application Domain scope).
Zero-Downtime Evolution Renaming or restructuring folders requires reloading master JCasC. Restructuring folders is applied dynamically via Git commit without touching master configuration.

3. Step-by-Step Technical Chain & Execution Flow

  1. JCasC Master Bootstrap:
    When the Jenkins container starts, jcasc/jenkins-jcasc.yaml configures the master Seed Job: 
    jobs:
  - script: |
      job('00-Seed-Job-BWCE-Platform-Orchestrator') {
        description('Master Seed Job provisioning TIBCO BWCE Pipelines as Code.')
        scm {
          git {
            remote { url('https://github.com/nubenetes/jenkins-git-parameter-bwce.git') }
            branch('*/main')
          }
        }
        steps {
          jobDsl {
            targets('jobdsl/seed-job.groovy\njobdsl/pipelines-ci.groovy\njobdsl/pipelines-cd.groovy')
            removedJobAction('DELETE')
            removedViewAction('DELETE')
            lookupStrategy('JENKINS_ROOT')
          }
        }
        triggers { scm('H/15 * * * *') }
      }
  2. lookupStrategy('JENKINS_ROOT') Context Resolution:
    By setting lookupStrategy('JENKINS_ROOT'), Job DSL resolves all relative job paths from the root of Jenkins. This allows pipelines-ci.groovy to target "01-CI-Build-Pipelines/${app.name}-ci-build" and nest jobs into the folders previously created by seed-job.groovy.
  3. Automated Drift Correction via SCM Polling:
    The trigger scm('H/15 * * * *') polls this repository every 15 minutes. When an engineer adds a new microservice to the def apps = [...] inventory in pipelines-ci.groovy and pushes to main, the Seed Job automatically generates the new CI/CD pipelines in Jenkins with zero manual UI configuration.

Pattern 1: Dual Git Parameter Dropdowns (Multi-Remote SCM)

Target: Developer Sandboxes & Feature Preview Environments

  • Dropdown 1: Queries application repository (APP_GIT_REVISION).
  • Dropdown 2: Queries global configuration repository (GLOBAL_VARS_REVISION).
  • Multi-remote refspecs configured via Job DSL in jobdsl/pipelines-ci.groovy.

Pattern 2: Decoupled CI Build & Multi-Cluster CD Orchestrator

Target: Enterprise Production Release Pipeline (Recommended)

🧪 Click to expand: Inner-Loop (Pattern 1) vs Outer-Loop (Pattern 2) Coexistence Diagram 
flowchart TD
    subgraph InnerLoop ["Pattern 1: Dual-Dropdown"]
        Dev1["Feature Dev"]
        DualJob["Dual-Dropdown Job<br/>App & Config Refs"]
        DevSandbox["Dev Sandbox<br/>dev-bwce"]

        Dev1 -->|"Trigger"| DualJob
        DualJob -->|"Deploy"| DevSandbox
    end

    subgraph OuterLoop ["Pattern 2: Decoupled CI/CD"]
        Dev2["Release Lead"]
        CIJob["01-CI-Build<br/>App Repo"]
        Registry["DEV Registry"]
        CDOrchestrator["02-CD-Orchestrator<br/>Global Vars SSOT"]
        ArgoEngine["ArgoCD Engine"]
        Clusters["Promotion Chain<br/>DEV->STG->PROD"]

        Dev2 -->|"Trigger CI"| CIJob
        CIJob -->|"Push Image"| Registry
        CIJob -->|"Dispatch"| CDOrchestrator
        CDOrchestrator -->|"Sync"| ArgoEngine
        ArgoEngine -->|"Promote"| Clusters
    end
Loading

📋 Dual-Pattern Coexistence Mechanics:

  • Inner-Loop (Pattern 1: Dual-Dropdown): Single pipeline for rapid BWCE developer testing, allowing instant pairing of feature branches with custom .substvar configurations.
  • Outer-Loop (Pattern 2: Decoupled Hand-off): Standardized enterprise release pipeline with independent CI compilation and governed multi-cluster CD promotion.

💡 Architectural Summary & Conclusion: Supporting both inner-loop and outer-loop patterns provides the agility needed during integration development alongside the governance required for production releases.

🔄 Click to expand: End-to-End Hand-off Sequence Diagram 
sequenceDiagram
    autonumber
    actor Dev as Developer / Operator
    participant CI as 01-CI-Build-Pipeline
    participant DevReg as OCP DEV Registry
    participant CD as 02-CD-Release-Orchestrator
    participant StgReg as OCP Staging Registry
    participant PrdReg as OCP Prod Registry
    participant Argo as ArgoCD 3.5 Control Plane
    participant DDog as Datadog Cloud APM

    Dev->>CI: Trigger Build (Select App Branch)
    activate CI
    CI->>CI: Execute BWUnit Mock Tests
    CI->>CI: Package EAR and Build Container
    CI->>CI: Scan with Trivy and Sign with Cosign
    CI->>DevReg: Push image (e.g. 2.1.0-42)
    CI->>DDog: Emit CI Span (ci.bwce.pipeline_completed)
    CI->>CD: Dispatch downstream CD Orchestrator
    deactivate CI

    activate CD
    CD->>Argo: Sync DEV Application (Wave 0 -> Wave 1)
    Argo-->>CD: DEV Health Check Passed (HTTP 200)
    CD->>StgReg: Skopeo copy (DEV -> STAGING)
    CD->>Argo: Sync STAGING Application
    Argo-->>CD: STAGING Integration Suite Passed (100%)
    
    CD->>Dev: Production Approval Gate (Interactive Input)
    Dev-->>CD: Approved by Release Engineer

    CD->>PrdReg: Skopeo copy (STAGING -> PROD)
    CD->>Argo: Sync PROD Application (Argo Rollouts Canary)
    Argo->>DDog: Query Datadog Metric Analysis (p99 < 250ms, 5xx < 0.1%)
    DDog-->>Argo: Metrics Passed (100% Promoted)
    CD->>DDog: Emit Release Metric (cd.bwce.release_completed)
    deactivate CD
Loading

📋 Step-by-Step CI/CD Hand-off Sequence:

  • 1. CI Pipeline Trigger: Launches Pipeline 01 with APP_GIT_REVISION.
  • 2. EAR Build & Test: Packages .ear via bw6-maven-plugin, runs BWUnit tests, layers onto tibco/bwce:2.9.2, and pushes to DEV registry.
  • 3. Downstream Hand-off: Triggers Pipeline 02 with IMAGE_TAG and target environment parameters.
  • 4. Promotion & Manifest Commit: Copies image via Skopeo and updates .substvar profile in jenkins-git-parameter-bwce-global-vars.
  • 5. ArgoCD Sync: Invokes argoAppSync to reconcile OpenShift clusters and verify HTTP 200 health.

💡 Architectural Summary & Conclusion: The decoupled CI/CD sequence enforces artifact immutability and ensures that promoted BWCE images are tested and verified before reaching production clusters.

Automated Downstream Hand-off & Environment Targeting for BWCE

In Pattern 2, Pipeline 01 (BWCE CI Build) and Pipeline 02 (CD Release Orchestrator) are connected via an automated downstream hand-off trigger:

🔄 Click to expand: BWCE Downstream Hand-off & Multi-Environment Promotion Flow Diagram 
flowchart LR
    subgraph CIStage["1. Pipeline 01: BWCE CI"]
        direction TB
        Dev1["👩‍💻 User"] -->|"Selects Branch & Env"| CI["🏗️ 01-CI-Build<br/>• Packages EAR<br/>• Signs Image<br/>• Pushes to DEV"]
        CI -->|"Trigger CD"| Dispatch["⚡ build job:<br/>02-CD-Orchestrator<br/>Passes: IMAGE_TAG"]
    end

    subgraph CDStage["2. Pipeline 02: CD Orchestrator"]
        direction TB
        Dispatch --> CD["🚀 02-CD-Orchestrator<br/>• Resolves .substvar<br/>• Routes by Env"]
        
        CD -->|"env == 'dev'"| DevTarget["☸️ OpenShift DEV<br/>• gitopsCommit(dev)<br/>• argoAppSync(dev)"]
        CD -->|"env == 'staging'"| StgTarget["☸️ OpenShift STAGING<br/>• skopeoPromote<br/>• argoAppSync(stg)"]
        CD -->|"env == 'prod'"| PrdTarget["☸️ OpenShift PROD<br/>• Approval Gate<br/>• skopeoPromote<br/>• argoAppSync(prd)"]
    end
Loading

📋 Downstream Parameter Passing & Execution Logic:

  • Configured in Job DSL (jobdsl/pipelines-ci.groovy):
    • TRIGGER_CD_RELEASE (boolean, default: true): Automatically dispatches downstream release.
    • GLOBAL_VARS_BRANCH (string, default: 'main'): Global configuration repository branch passed to Pipeline 02.
    • TARGET_ENVIRONMENT (choice: dev, staging, prod, full-promotion-chain): Initial target environment.
  • Trigger Stage in Pipeline 01 (Jenkinsfile.app-bwce): 
    stage('Trigger CD Release Orchestrator') {
    when { expression { return params.TRIGGER_CD_RELEASE == true } }
    steps {
        build job: '02-CD-Release-Orchestrators/multi-cluster-release-orchestrator',
            parameters: [
                string(name: 'GLOBAL_VARS_REVISION', value: params.GLOBAL_VARS_BRANCH ?: 'main'),
                string(name: 'APP_NAME', value: env.APP_NAME),
                string(name: 'IMAGE_TAG', value: env.CALCULATED_TAG),
                string(name: 'TARGET_ENVIRONMENT', value: params.TARGET_ENVIRONMENT ?: 'dev'),
                booleanParam(name: 'AUTO_PROMOTE_TO_STAGING', value: true),
                booleanParam(name: 'REQUIRE_PROD_APPROVAL', value: true),
                string(name: 'TRIGGERED_BY', value: 'CI_PIPELINE'),
                string(name: 'CHANGE_REQUEST_ID', value: "CI-BWCE-AUTO-${BUILD_NUMBER}")
            ],
            wait: false
    }
}
  • Environment-Specific Routing in Pipeline 02 (Jenkinsfile.release-orchestrator):
    • Target dev: Binds DEV.substvar profile and triggers argoAppSync on OpenShift DEV.
    • Target staging: Promotes container image via Skopeo, binds STAGING.substvar, and synchronizes OpenShift STAGING.
    • Target prod: Promotes image to PROD registry, pauses at the interactive Production Approval Gate (input step), binds PROD.substvar, and deploys via ArgoCD canary sync waves on OpenShift PROD.

💡 Architectural Summary & Conclusion: This automated linkage guarantees 'Build Once, Deploy Anywhere' artifact immutability for TIBCO BWCE. The EAR archive is compiled and packaged once in CI, while Pipeline 02 promotes the verified container image across environments by binding the appropriate .substvar profiles.

Enterprise Security & Supply Chain Integrity

1. Cosign Image Signing & SLSA Level 3 Attestation

All container images layered on the BWCE base runtime are cryptographically signed using Sigstore Cosign during CI execution, guaranteeing supply chain integrity before promotion into production.

🔏 Click to expand: Cosign Image Signing & SLSA Level 3 Attestation Flow Diagram 
flowchart LR
    CI["CI Build<br/>(Maven BWCE)"]
    Trivy["Trivy Scan<br/>& Syft SBOM"]
    Cosign["Cosign Sign<br/>& Attestation"]
    Registry["OCP DEV<br/>Registry"]
    Policy["OpenShift Policy<br/>Block Unsigned"]

    CI --> Trivy
    Trivy --> Cosign
    Cosign --> Registry
    Registry -.->|"Enforced"| Policy
Loading

📋 Supply Chain Security & Signing Chain:

  • Artifact Packaging: Compiles EAR and builds container image.
  • Cosign Signing: Ephemeral agent signs the image digest using keys stored in HashiCorp Vault.
  • Attestation Generation: Attaches CycloneDX SBOM and SLSA provenance to the OpenShift image registry.
  • Admission Verification: OpenShift Image Signature Policy enforces signature verification at pod admission.

💡 Architectural Summary & Conclusion: SLSA Level 3 supply chain security ensures that only authenticated, signed BWCE container images can be deployed across enterprise clusters.

2. Zero-Trust Secrets with External Secrets Operator & Vault

Plaintext credentials (database passwords, JMS connection tokens, Datadog API keys) are never stored in Git. Instead, the External Secrets Operator (ESO) synchronizes encrypted secrets from HashiCorp Vault directly into OpenShift namespaces.

🔐 Click to expand: External Secrets Operator & Vault Synchronization Flow Diagram 
flowchart LR
    Vault["HashiCorp Vault<br/>Secrets & Keys"]
    GlobalVars["Global Vars Repo<br/>ExternalSecret CRD"]
    ESO["External Secrets<br/>Operator (ESO)"]
    K8sSecret["Kubernetes Secret<br/>(In-Memory)"]
    BWCEPod["TIBCO BWCE Pod<br/>(Workload)"]

    Vault -->|"Token Fetch"| ESO
    GlobalVars -->|"Ref Spec"| ESO
    ESO -->|"Sync/Rotate"| K8sSecret
    K8sSecret -->|"Env Mount"| BWCEPod
Loading

📋 Secret Management & Dynamic Injection:

  • Centralized Storage: HashiCorp Vault securely stores backend database credentials, keystores, and API tokens.
  • External Secrets Operator (ESO): Synchronizes secrets into native Kubernetes Secret resources.
  • Dynamic Injection: BWCE runtime consumes secrets at startup without hardcoding sensitive data in .substvar files.

💡 Architectural Summary & Conclusion: ESO with HashiCorp Vault eliminates hardcoded passwords from BWCE property files, enabling centralized rotation and enterprise audit compliance.

3. Ephemeral PR Preview Environments via ArgoCD ApplicationSets

Pull Requests targeting application repositories automatically trigger the ArgoCD PR Generator ApplicationSet, creating ephemeral preview namespaces for end-to-end integration testing before merging.

⚡ Click to expand: Ephemeral PR Preview Provisioning Flow Diagram 
flowchart LR
    Dev["Developer"]
    PR["GitHub PR #42"]
    AppSet["ArgoCD AppSet<br/>(PR Generator)"]
    PreviewNS["Preview Env<br/>pr-42-preview"]
    Datadog["Datadog<br/>PR Dashboard"]

    Dev -->|"Opens"| PR
    PR -->|"Discovers"| AppSet
    AppSet -->|"Deploys"| PreviewNS
    PreviewNS -->|"DD_ENV"| Datadog
Loading

📋 Dynamic Ephemeral Preview Provisioning Flow:

  • PR Trigger: Opening a pull request triggers Jenkins CI to package an ephemeral BWCE preview image.
  • ArgoCD Dynamic Sync: Provisions an isolated namespace pr-preview-<id>-bwce on DEV cluster.
  • Automated Cleanup: Closing or merging the pull request automatically deletes the namespace and temporary resources.

💡 Architectural Summary & Conclusion: Ephemeral preview environments provide isolated sandboxes for validating BWCE business logic and API contracts before merging changes to main branches.

📂 Repository Structure

.
├── config/
│   ├── environments.env              # Pinned versions, domain names, Datadog credentials
│   └── clusters.yaml                 # ConfigMap defining OCP DEV, STAGING, PROD topologies
├── helm/
│   ├── jenkins/                      # Official Jenkins chart values with Datadog plugin
│   ├── observability/                # Datadog Agent Helm values (OpenShift hardened)
│   └── argocd/                       # ArgoCD 3.5 Helm values
├── jcasc/
│   ├── jenkins-jcasc.yaml            # JCasC security matrix, Datadog CI visibility, seed job
│   ├── pod-templates.yaml            # Ephemeral Kubernetes agent pod templates (bwce-builder)
│   └── github-app-credentials.yaml   # GitHub App credential rotation
├── jobdsl/
│   ├── seed-job.groovy               # Master Seed Job definition
│   ├── pipelines-ci.groovy           # Pattern 1 and Pattern 2 CI pipelines
│   └── pipelines-cd.groovy           # CD Release Orchestrator & Hotfix pipelines
├── jenkinsfiles/
│   ├── ci/
│   │   ├── Jenkinsfile.app-bwce
│   │   └── Jenkinsfile.app-bwce-dual-dropdown
│   └── cd/
│       ├── Jenkinsfile.release-orchestrator
│       └── Jenkinsfile.hotfix-deploy
├── shared-library/
│   ├── vars/
│   │   ├── datadogLogEvent.groovy    # Emits Datadog CI spans & DogStatsD metrics
│   │   ├── bwceEarBuild.groovy       # Compiles and packages BWCE EAR
│   │   ├── bwceProfileOverride.groovy# Injects .substvar profile tokens
│   │   ├── argoAppSync.groovy        # ArgoCD 3.5 synchronization
│   │   ├── skopeoPromote.groovy      # Cross-cluster image promotion
│   │   ├── cosignSign.groovy         # Cryptographic image signing (SLSA Level 3)
│   │   ├── sbomGenerate.groovy       # CycloneDX SBOM generation
│   │   └── gitopsCommit.groovy       # Updates Global Vars SSOT repository
│   └── src/com/nubenetes/gitops/
├── sample-apps/
│   ├── tibco-bwce-order-service/     # Complete TIBCO BWCE project + Dockerfile + K8s + Rollout
│   └── tibco-bwce-customer-api/
├── observability/
│   ├── dashboards/                   # Datadog dashboards (Jenkins, BWCE Engine, ArgoCD)
│   └── monitors/                     # Datadog automated alert monitors
├── argocd-apps/                      # ArgoCD Root App-of-Apps & ApplicationSets
├── security/                         # Image signature policy & External Secrets Operator
├── scripts/                          # Automation & setup scripts
├── deploy.sh                         # 1-Click Platform Deployment Script
├── destroy.sh                        # Clean Decommission Script
├── reinstall.sh                      # Full Wipe & Reinstall Script
└── Makefile

🚀 Quick Start & Deployment

1. Prerequisites

  • Red Hat OpenShift 4.20+ cluster (or Kubernetes 1.31+)
  • kubectl, oc, and helm v3 CLI tools installed
  • Datadog Account & API/App Keys

2. Configure Credentials

Set your Datadog credentials in config/environments.env or export environment variables:

export DATADOG_API_KEY="your-datadog-api-key"
export DATADOG_APP_KEY="your-datadog-app-key"

3. Deploy Platform

./deploy.sh
# or
make deploy

4. Access Platform Services

  • Jenkins Controller: https://jenkins-jenkins.apps.ocp-dev.nubenetes.internal (Default login: admin / admin123!)
  • ArgoCD UI: https://argocd-server.apps.ocp-dev.nubenetes.internal
  • Datadog Dashboard: https://app.datadoghq.eu

Decommissioning & Reinstallation

Clean Decommission

./destroy.sh
# or
make destroy

Full Reinstallation

./reinstall.sh
# or
make reinstall

References & Standards

  1. TIBCO BWCE Documentation: TIBCO BusinessWorks™ Container Edition Official Docs
  2. Datadog Jenkins Integration: Datadog CI Visibility & Jenkins Plugin
  3. Datadog Java Tracer APM: Datadog Java Agent Documentation
  4. ArgoCD 3.5 & ApplicationSets: ArgoCD Documentation
  5. Sigstore Cosign: Cosign Container Signing Documentation
  6. External Secrets Operator: External Secrets Operator Docs
  7. Jenkins Job DSL & Git Parameter: Jenkins Job DSL API

🔗 Related Repositories

About

Enterprise Jenkins CI/CD & ArgoCD GitOps for TIBCO BWCE + Datadog APM | Paired with nubenetes/jenkins-git-parameter-bwce-global-vars

github.com/nubenetes/jenkins-git-parameter-bwce-global-vars

Topics

argo-rolloutsargocdbwcedatadogdatadog-apmdevopsgitopsjcascjenkinsjob-dslkubernetesopenshiftslsa-level-3tibcotibco-bwce

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages