Important
- Initial Generation: Originally generated by Gemini 3.7 Flash with Antigravity.
- Review & Production Hardening: Audited, remediated, and verified for enterprise readiness using Gemini 3.8 Flash with Antigravity.
- Testing Status: This repository serves as an architectural reference blueprint, educational design pattern, and foundational boilerplate. Platform engineers must review and validate all cluster endpoints, secrets, and TLS certificates prior to live production deployment.
Important
This platform implements the Pure GitOps (Pull-based) Architecture, serving as the modern cloud-native alternative to the push-based nubenetes/jenkins-git-parameter pattern:
- 🚀 Pure GitOps Platform Orchestrator (This Repository):
nubenetes/jenkins-without-git-parameter— Infrastructure-as-Code, Lean Jenkins JCasC CI, ArgoCD 3.5 ApplicationSets, Native TargetRevision Selection, and OpenTelemetry Observability. - 🌐 Reference Push-Based Repository:
nubenetes/jenkins-git-parameter— Legacy/Push model with Jenkins UIgitParameterdropdowns and multi-remote SCM Job DSL. - ☕ Embedded Workload Microservice: Located directly within
sample-apps/jhipster-microservice— Java 21 / Spring Boot 3 cloud-native microservice with OpenTelemetry and Prometheus integration.
- Executive Summary & Architectural Paradigm Shift
- In-Depth Architectural Comparison: Push vs. Pull Model
- 1. Comprehensive Comparison Matrix: Jenkins Git Parameter vs. Pure GitOps
- 2. The Root Cause of Jenkins SCM Friction (Why Git Parameter Fails at Scale)
- 3. How Git & ArgoCD Solve Parameterization Natively
- 4. Why There is No 'jenkins-without-git-parameter-global-vars' Repository (Architectural Rationale)
- 5. Decoupled Architecture: Docker Images vs. Environment Variables & Placeholders
- 6. Enterprise Developer Portals & Governance: Backstage IDP & ServiceNow / Jira ITSM Integration
- Comprehensive Mermaid Architecture Diagrams & Workflows
- 1. End-to-End Multi-Cluster Platform Topology
- 2. Jenkins SCM Pre-Execution Lifecycle Blindspot
- 3. Side-by-Side Flow Comparison: Push vs. Pull
- 4. Dynamic Ephemeral Pull Request (PR) Preview Environments
- 5. Automated CI -> GitOps Promotion Sequence
- 6. ArgoCD Multi-Cluster Matrix Reconciliation Engine
- 7. Zero-Trust Security & RBAC Boundary Architecture
- 8. Progressive Delivery with Argo Rollouts Canary
- 9. Full-Stack Observability & Trace Context Propagation
- Which Pattern is Easier, Recommended, and Why?
- Platform Architecture & Component Details
- Repository Structure
- Quick Start: 1-Click Automated Deployment
- Decommissioning & Reinstallation
- References & Evidence Links
In modern cloud-native engineering on Red Hat OpenShift 4.20+, organizations face a pivotal architectural choice for continuous delivery:
-
The Imperative Push Model (
jenkins-git-parameter): Jenkins is treated as a monolithic orchestrator. Developers open the Jenkins Web UI, select branches, git tags, or commit SHAs via thegit-parameterplugin, and Jenkins pushes container images and executes imperative deployment commands (oc apply,helm upgrade, orargocd app sync) directly against target clusters. -
The Declarative Pull Model (
jenkins-without-git-parameter- Pure GitOps): Jenkins is decoupled and strictly scoped to Continuous Integration (CI) (linting, automated testing, container image packaging, Trivy vulnerability scanning, Syft SBOM generation, and Cosign SLSA 3 signing). Jenkins has zero deployment parameters and zero cluster credentials.All parameter selections (branches, tags, commits, environment overrides) are executed natively via Git (the Single Source of Truth) and ArgoCD 3.5:
- Branch & commit selection via Git Pull Requests and merge workflows.
- Release selection via Git Semantic Versioning tags (
v1.2.0,release-2026.09). - Dynamic ephemeral preview environments via ArgoCD ApplicationSets (PR and Branch Generators).
- On-demand revision overrides via ArgoCD UI/CLI native
targetRevisiontracking.
This repository provides the complete, production-ready Infrastructure as Code (IaC) blueprint, JCasC, Multibranch Pipelines, Helm values, ArgoCD ApplicationSets, and OpenTelemetry observability configurations implementing the Pure GitOps pattern.
Important
Context & Heritage: The Patterns in jenkins-git-parameter vs. Pure GitOps
In the baseline repository jenkins-git-parameter, we investigated and delivered two core Jenkins push patterns for multi-repository environments:
- Pattern 1: Dual Git Parameter Dropdowns in a Single Pipeline (Multi-Remote SCM):
Configured both the application code repo andjenkins-git-parameter-global-varsinside a singleJob DSLdefinition using custom refspecs (origin-appandorigin-vars). While functional, it suffered from SCM namespace collisions, Jenkins master UI render latency, and workspace checkout quirks. - Pattern 2: Decoupled Two-Pipeline Architecture (CI ➔ CD Hand-off) [RECOMMENDED in
jenkins-git-parameter]:
Separated artifact creation from release promotion:- Pipeline 01 (
01-CI-Build-Pipelines/*-ci-build): Bound to the application code repo with anAPP_GIT_REVISIONdropdown. Built the container image once, ran tests, and passed the image tag downstream. - Pipeline 02 (
02-CD-Release-Orchestrators/multi-cluster-release-orchestrator): Bound directly tojenkins-git-parameter-global-varswith its ownGLOBAL_VARS_REVISIONdropdown (DEV,STAGING,PROD). Orchestrated multi-cluster image promotion (via Skopeo), committed updated tags to GitOps manifests, and invokedargoAppSync.
- Pipeline 01 (
While Pattern 2 was the best possible design within the Jenkins Push Paradigm (honoring Single Responsibility Principle and "Build Once, Deploy Anywhere"), it still kept Jenkins as the manual Parameter Proxy, release coordinator, and credential holder in Pipeline 02.
In jenkins-without-git-parameter (This Repository), we take the architectural evolution to its ultimate cloud-native conclusion:
- Pipeline 02 is completely eliminated from Jenkins: ArgoCD 3.5 natively performs all continuous delivery, drift detection, and multi-cluster reconciliation directly from Git.
- Pipeline 01 becomes a parameterless, webhook-driven CI engine: Developers never interact with Jenkins UI dropdowns; pushing code or opening PRs automatically builds, scans (Trivy), generates SBOM (Syft), signs (Cosign SLSA 3), and commits the image tag to GitOps.
- Release selection is native to Git & ArgoCD: Handled via Git PRs, Semantic Version tags (
v1.2.0), or ArgoCD's nativetargetRevisionUI/CLI.
Click to expand: 🔄 Three-Way Architectural Evolution Diagram (Pattern 1 vs. Pattern 2 vs. Pure GitOps)
flowchart TB
subgraph P1["Pattern 1: Push"]
direction TB
Dev1["👩💻 User"] -->|"1. Select Dropdowns"| JJob1["📋 Single Pipeline<br/>• Multi-Remote SCM<br/>• origin-app<br/>• origin-vars"]
JJob1 -->|"2. Builds & Deploys"| Agent1["⚙️ Jenkins Agent<br/>• Cluster Secrets"]
Agent1 -->|"3. Imperative Push"| K8s1["☸️ OpenShift Clusters"]
end
subgraph P2["Pattern 2: Hand-off"]
direction TB
Dev2["👩💻 User"] -->|"1. Selects App Branch"| CI2["🏗️ Pipeline 01: CI<br/>• App Git Parameter"]
CI2 -->|"2. Builds Image Once"| Reg2["🐳 Container Registry"]
CI2 -->|"3. Triggers Downstream"| CD2["🚀 Pipeline 02: CD<br/>• Global Vars Dropdown<br/>• Release Orchestrator"]
CD2 -->|"4. Skopeo & Commit"| GitOps2["🌐 GitOps Repo<br/>(global-vars)"]
CD2 -->|"5. Calls argoAppSync"| Argo2["🐙 ArgoCD Controller"]
Argo2 -->|"6. Syncs Cluster"| K8s2["☸️ OpenShift Clusters"]
end
subgraph P3["Pattern 3: Pure GitOps"]
direction TB
Dev3["👩💻 Developer"] -->|"1. Git PR / Tag"| Git3["🐙 Git Repo (SSOT)<br/>• App Code<br/>• GitOps Overlays"]
Git3 -.->|"2. Webhook Event"| CI3["🏗️ Lean Jenkins CI<br/>• Zero UI Params<br/>• Multibranch Webhook"]
CI3 -->|"3. Build & Scan"| Reg3["🐳 Container Registry<br/>(Cosign SLSA 3)"]
CI3 -->|"4. Auto-commit Tag"| Git3
Git3 -->|"5. Continuous Sync"| Argo3["🐙 ArgoCD 3.5 Engine<br/>• targetRevision<br/>• ApplicationSets"]
Argo3 -->|"6. Self-Healing"| K8s3["☸️ OpenShift Runtime<br/>• Zero Secrets in CI<br/>• Multi-Cluster"]
end
| Architectural Feature | Pattern 1 (Dual Dropdown Push) | Pattern 2 (Decoupled CI ➔ CD Hand-off Push) | Pure GitOps (This Repository) |
|---|---|---|---|
| Jenkins Jobs Count | 1 Monolithic Pipeline per app. | 2 Pipelines (01-CI-Build + 02-CD-Orchestrator). | 1 Parameterless CI Pipeline per app. |
| Parameter Interface | Jenkins UI (2 dropdowns in 1 job). | Jenkins UI (1 dropdown in CI, 1 dropdown in CD). | Git (PRs/Tags) & ArgoCD targetRevision. |
| SCM Complexity | High (Multi-remote refspec bindings). | Moderate (Isolated SCM bindings per pipeline). | Zero SCM Hacks (Standard webhooks/branches). |
| Release Trigger | Manual human click in Jenkins. | Manual click or downstream trigger to Pipeline 02. | 100% Event-Driven via Git webhooks / PRs. |
| Jenkins Credentials | High (Cluster tokens & Git write). | High (Cluster tokens, Skopeo, ArgoCD API keys). | Zero Cluster Credentials (Least Privilege). |
| Deployment Engine | Jenkins Agent (Push). | Jenkins Agent ➔ invokes ArgoCD sync. | ArgoCD 3.5 Pull Controller (Continuous). |
| Ephemeral PR Envs | Complex Groovy scripts. | Complex Groovy scripts in Pipeline 02. | Native ArgoCD ApplicationSet PR Generator. |
| Configuration Drift | Blindspot (No self-healing). | Blindspot (ArgoCD only syncs when Jenkins runs). | Continuous Self-Healing (24/7 reconciliation). |
💡 Architectural Summary & Conclusion: While Pattern 2 in
jenkins-git-parameterestablished best practices within the push ecosystem by separating build and release pipelines, Pure GitOps completes the cloud-native transition by eliminating Jenkins from the release path entirely. By shifting parameter selection to native Git pull requests and ArgoCD controllers, the platform achieves zero-trust isolation, removes all cluster credentials from CI, and enables continuous 24/7 self-healing.
A critical architectural distinction is: Why does Pure GitOps completely eliminate Pipeline 02 from Jenkins rather than chaining downstream pipelines?
Click to expand: 🔄 Pattern 2 Downstream Push vs. Pure GitOps Event-Driven Pull Diagram
flowchart TB
subgraph LegacyPush["1. Pattern 2: Push"]
direction TB
CI1["🏗️ Pipeline 01: CI<br/>• Builds Image<br/>• Passes Tag"] -->|"build job: '02-CD...'<br/>(Passes: TARGET_ENV)"| CD1["🚀 Pipeline 02: CD<br/>• Release Lead<br/>• Cluster Secrets"]
CD1 -->|"argoAppSync / Skopeo"| Cluster1["☸️ OpenShift (Push)<br/>(DEV / STG / PROD)"]
end
subgraph PureGitOps["2. Pure GitOps"]
direction TB
CI2["🏗️ Lean CI (Pull)<br/>• Builds & Signs<br/>• Single Pipeline"] -->|"gitopsCommit (Git)"| GitSSOT["🐙 Git Repo (SSOT)<br/>• Overlays<br/>• Image Digests"]
GitSSOT -->|"Continuous Sync"| ArgoCD["🐙 ArgoCD 3.5 Engine<br/>• Reconciles State<br/>• Zero CI Secrets"]
ArgoCD -->|"Declarative Pull"| Cluster2["☸️ OpenShift (Pull)<br/>(DEV / STG / PROD)"]
end
| Architectural Dimension | jenkins-git-parameter (Pattern 2) |
jenkins-without-git-parameter (Pure GitOps) |
|---|---|---|
| Pipelines in Jenkins | 2 Pipelines: 01-ci-build + 02-release-orchestrator. |
1 Pipeline: 01-CI-Build-Pipelines (Lean Multibranch). |
| Downstream Linkage | Jenkins build job: '02-CD-Release-Orchestrators/...'. |
No downstream Jenkins job. CI commits image tag to Git and terminates. |
| Who Orchestrates CD? | Jenkins Pipeline 02 (promotes via Skopeo, calls argoAppSync). |
ArgoCD 3.5 Controller (tracks Git and reconciles clusters continuously). |
| Environment Targeting | Chosen in Jenkins parameter dropdown (TARGET_ENVIRONMENT). |
Driven by Git Branch / PR: main ➔ DEV, PR/Tag ➔ STAGING/PROD. |
| Promotion Mechanism | Pipeline 02 promotes via Skopeo and runs approval gates (input). |
Git Pull Request (PR): Merging a PR into environment manifests triggers ArgoCD. |
| Cluster Credentials | Stored inside Jenkins agents (argocd-gitops). |
Zero cluster credentials in Jenkins. Only ArgoCD accesses OpenShift. |
In jenkinsfiles/ci/Jenkinsfile.app-java-maven, the pipeline ends immediately after signing the container image and writing the digest to Git—completely removing downstream job invocations:
stage('GitOps Automatic Sync (Update Git Repository)') {
steps {
script {
def targetEnv = (env.BRANCH_NAME == 'main') ? 'dev' : ((env.BRANCH_NAME == 'staging' || env.TAG_NAME) ? 'staging' : 'dev')
echo "📝 Updating GitOps Repository for environment: ${targetEnv} with new image tag: ${env.IMAGE_TAG}"
// Jenkins writes directly to Git, then terminates.
gitopsCommit(
appName: env.APP_NAME,
imageTag: env.IMAGE_TAG,
envName: targetEnv,
commitMsg: "chore(gitops): promote ${env.APP_NAME} to ${env.IMAGE_TAG} for ${targetEnv} [skip ci]"
)
}
}
}💡 Architectural Summary & Conclusion: Retiring Pipeline 02 completes the cloud-native transition. Jenkins is restored to its core strength as a fast, isolated CI build and test engine, while ArgoCD assumes full ownership of deployment orchestration, continuous drift correction, and zero-trust multi-cluster delivery.
| Architectural Dimension | jenkins-git-parameter (Imperative Push) |
jenkins-without-git-parameter (Declarative Pull / Pure GitOps) |
Advantage & Recommendation |
|---|---|---|---|
| Architectural Paradigm | Push: Jenkins acts as the central master driving both build and cluster state. | Pull: Jenkins does CI only; ArgoCD continuously pulls and reconciles cluster state from Git. | 🏆 GitOps: Cloud-native standard (CNCF / OpenGitOps). |
| Where Parameters are Selected | Jenkins Web UI: User selects branch/tag/commit via git-parameter dropdowns. |
Git & ArgoCD: Selected via Git branches, tags, PRs, or ArgoCD targetRevision UI/CLI. |
🏆 GitOps: Eliminates Jenkins UI lag and human form errors. |
| Single Source of Truth (SSOT) | Split/Ambiguous: Jenkins build logs and job history hold the state of what was deployed. | Pure Git: Git repository commit history is the single, cryptographically auditable source of truth. | 🏆 GitOps: Immutable commit logs with GPG/Cosign attestation. |
| Jenkins Master Performance | High Overhead: Master dynamically queries remote Git refs before rendering build forms; prone to SCM latency. | Zero Overhead: Jenkins uses lightweight webhooks and multibranch indexing; zero UI querying overhead. | 🏆 GitOps: Controller remains fast, stable, and highly responsive. |
| Multi-Repository Complexity | Complex & Fragile: Requires multi-remote SCM hacks in Job DSL (origin-app, origin-vars) to query two repos. |
Decoupled & Native: Application code and GitOps manifests are cleanly separated; ArgoCD manages multi-repo natively. | 🏆 GitOps: Clean separation of concerns (Code vs Config). |
| Security & Credential Boundary | High Exposure: Jenkins Master and ephemeral agents require cluster admin tokens and ArgoCD API keys. | Zero-Trust: Jenkins only has write access to internal container registry & Git repo. Only ArgoCD has cluster access. | 🏆 GitOps: Drastically reduced blast radius and attack surface. |
| Configuration Drift Handling | Blindspot: If someone changes a resource via oc edit or kubectl, Jenkins cannot detect or fix it until the next build. |
Continuous Self-Healing: ArgoCD continuously monitors cluster state and automatically self-heals any unauthorized drift. | 🏆 GitOps: Zero drift guarantee across all OpenShift clusters. |
| Rollback Mechanism | Imperative Re-run: Requires finding a prior Jenkins build ID, selecting the old git tag, and running the pipeline. | Instant Git Revert: Standard git revert <commit> or 1-click rollback in ArgoCD UI/CLI. |
🏆 GitOps: Deterministic, instant, and fully traceable. |
| Ephemeral / Preview Environments | Script-Heavy: Jenkinsfiles must contain complex Groovy stages to oc create ns, deploy, and register cleanup hooks. |
Declarative ApplicationSets: ArgoCD PR Generator automatically creates and destroys preview environments per GitHub PR. | 🏆 GitOps: Zero custom Groovy scripts; 100% automated lifecycle. |
| Multi-Cluster Scalability | Difficult: Jenkins pipelines must iterate through cluster endpoints sequentially with individual credentials. | Native Matrix Generator: ArgoCD ApplicationSet Matrix Generator deploys to 100+ clusters via simple cluster label matching. | 🏆 GitOps: Infinite horizontal cluster scalability. |
| Disaster Recovery (DR) | Time-Consuming: Must restore Jenkins database, job histories, and rerun pipelines in order. | Instantaneous: Point ArgoCD to the Git repository in a new cluster, and the entire state is re-created in minutes. | 🏆 GitOps: Complete disaster recovery in a single kubectl apply. |
| Plugin Maintenance & Vulnerabilities | High Maintenance: Requires git-parameter, active-choices, matrix-project, and SCM plugins that need frequent patching. |
Lean & Resilient: Zero parameter plugins needed. Jenkins runs with core pipeline and branch source plugins. | 🏆 GitOps: Minimal plugin footprint, fewer CVEs, easier upgrades. |
In the jenkins-git-parameter pattern, engineering teams hit three major architectural bottlenecks:
- The Pre-Execution vs. Runtime Lifecycle Paradox:
Jenkins renders build parameter dropdowns in the browser before launching an agent pod, before cloning source code, and before running pipeline stages. Therefore,
git-parametercan only query Git repositories statically defined in the Jenkins Master's Job XML configuration. - The SCM Multi-Remote Binding Bottleneck:
Declarative Pipelines (
cpsScm) were designed for a single primary SCM. When a pipeline needs to select a branch from an application repository (app-repo) AND a configuration tag from an environment repository (global-vars), Job DSL must configure multi-remote Git refspecs (origin-app,origin-vars), making jobs brittle and complex. - Security Inversion: In push-based Jenkins, Jenkins agents must possess broad OpenShift cluster administrative privileges or ArgoCD admin tokens to deploy resources, violating the principle of least privilege.
By transitioning to the Pure GitOps pattern (jenkins-without-git-parameter), parameterization is decoupled from the CI server and moved to native Git and ArgoCD primitives:
┌──────────────────────────────────────────────────────────────────────────────────────────┐
│ Pure GitOps Parameter Selection │
├──────────────────────────────────────┬───────────────────────────────────────────────────┤
│ Requirement │ How it is Handled in Pure GitOps │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 1. Deploying a Feature Branch │ Open a Pull Request in GitHub ──> │
│ │ ArgoCD ApplicationSet PR Generator creates │
│ │ preview-pr-<number> environment automatically. │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 2. Deploying a Release Tag (v1.2.0) │ Create a Git Release Tag `v1.2.0` in GitOps repo │
│ │ (or update targetRevision in ArgoCD Application). │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 3. Ad-hoc Commit / Hotfix Override │ Execute `argocd app set <app> --revision <sha>` │
│ │ or select revision in the ArgoCD Web UI dialog. │
├──────────────────────────────────────┼───────────────────────────────────────────────────┤
│ 4. Promoting DEV -> STAGING -> PROD │ Jenkins CI automatically commits image tag to Git │
│ │ or opens a Promotion PR across environment files. │
└──────────────────────────────────────┴───────────────────────────────────────────────────┘
A frequent question when transitioning from the legacy jenkins-git-parameter pattern to pure GitOps is: "Why is there no jenkins-without-git-parameter-global-vars repository?"
Click to expand: 🌐 Legacy Global-Vars vs. Pure GitOps Repository Topology Diagram
flowchart TB
subgraph LegacyModel["1. Legacy Push Model"]
direction TB
AppRepo1["📦 App Code Repo<br/>(sample-apps)"]
GlobalVars1["🌐 Global Vars Repo<br/>(global-vars)"]
Jenkins1["⚙️ Jenkins Master<br/>• Dropdown 1: App<br/>• Dropdown 2: Vars"]
Cluster1["☸️ OpenShift Clusters<br/>(DEV / STG / PROD)"]
AppRepo1 --> Jenkins1
GlobalVars1 --> Jenkins1
Jenkins1 -->|"Imperative Push"| Cluster1
end
subgraph GitOpsModel["2. Pure GitOps Model"]
direction TB
AppRepo2["📦 App Code Repo<br/>(sample-apps)"]
Jenkins2["🏗️ Lean Jenkins CI<br/>• Builds image<br/>• Signs (SLSA 3)<br/>• Auto-commits tag"]
GitOpsRepo["🌐 GitOps Repo (SSOT)<br/>• clusters.yaml<br/>• overlays:<br/>(dev, stg, prod)"]
ArgoCD["🐙 ArgoCD 3.5 Engine<br/>(Continuous Sync)"]
Cluster2["☸️ OpenShift Clusters<br/>(DEV / STG / PROD)"]
AppRepo2 -.->|"Webhook"| Jenkins2
Jenkins2 -->|"Auto-commits Tag"| GitOpsRepo
GitOpsRepo -->|"Continuous Sync"| ArgoCD
ArgoCD -->|"Declarative Pull"| Cluster2
end
- Jenkins No Longer Has UI Dropdowns for Deployment:
- In
jenkins-git-parameter, the user opened the Jenkins Web UI form and selected two dropdown parameters:APP_REVISION(microservice branch) andGLOBAL_VARS_REVISION(environment config branch/tag). - In
jenkins-without-git-parameter, Jenkins CI runs 100% automated via webhooks. It builds and tests the commit, pushes the container image, and writes the new image tag to Git. There is no human in Jenkins selecting a configuration version, so Jenkins has no need for a dedicated configuration dropdown repo.
- In
- Configuration Belongs to ArgoCD (GitOps), Not to Jenkins:
- In pure GitOps, environment definitions, cluster inventories, and Helm/Kustomize values belong to the GitOps Control Plane (ArgoCD).
- Naming a configuration repository
jenkins-*-global-varsis an anti-pattern in modern cloud-native architectures because Jenkins does not own the cluster state—ArgoCD and Git own it.
- Elimination of the SCM Multi-Remote Workaround:
- In Jenkins, querying two git repositories in a single declarative pipeline required complex Job DSL refspec workarounds (
origin-app,origin-vars). - In Pure GitOps, this problem completely vanishes. Application source code and deployment manifests are decoupled by design.
- In Jenkins, querying two git repositories in a single declarative pipeline required complex Job DSL refspec workarounds (
- Self-Contained GitOps Structure:
- All cluster definitions (
config/clusters.yaml), environment overlays (sample-apps/gitops-manifests/environments/), and application definitions (argocd-apps/) are cleanly organized within standard GitOps structures without requiring Jenkins-specific bindings.
- All cluster definitions (
| Feature | Legacy jenkins-git-parameter-global-vars |
GitOps Manifests (jenkins-without-git-parameter) |
|---|---|---|
| Primary Consumer | Jenkins Master (queries refs for UI dropdown). | ArgoCD 3.5 (reconciles cluster state continuously). |
| Parameter Mechanism | Jenkins gitParameter plugin & Active Choices. |
Git branches, release tags (v1.2.0), or PRs. |
| Who Modifies It? | Developers manually via Jenkins UI inputs. | Automated Bot: Jenkins CI auto-commits image tags on merge. |
| Drift Detection | None (Jenkins cannot detect cluster drift). | Continuous: ArgoCD self-heals any divergence from Git. |
| Repo Maintenance | High (must synchronize Jenkins credentials & SCM). | Low (Standard Git repository tracked by ArgoCD). |
If your organization prefers separating the platform IaC from workload manifests, there are two standard topologies:
- Platform Monorepo (Current Repository):
- Everything (Helm charts, JCasC, ArgoCD ApplicationSets, and sample GitOps manifests) lives in
jenkins-without-git-parameter. - Best for: 1-click deployment, self-contained demos, and small-to-medium platforms.
- Everything (Helm charts, JCasC, ArgoCD ApplicationSets, and sample GitOps manifests) lives in
- Enterprise Multi-Repo (Polyrepo GitOps):
app-repo(e.g.,jhipster-microservice): Developer source code + Dockerfile.ci-platform-repo(jenkins-without-git-parameter): Jenkins JCasC, Helm values, and Job DSL.gitops-manifests-repo(e.g.,nubenetes/gitops-manifests): Pure Kubernetes/ArgoCD environment overlays (dev,staging,prod).- Notice that even in a polyrepo setup, the repository is named
gitops-manifests, neverjenkins-global-vars, because it is managed by ArgoCD, not Jenkins.
💡 Architectural Summary & Conclusion: Separate
*-global-varsrepositories were a workaround required only because Jenkins used manual UI parameter forms to select multi-repository branch combinations. In Pure GitOps, configuration belongs natively to ArgoCD and Git manifests (k8s/overlays/), eliminating cross-repository coordination overhead and ensuring that all platform assets remain atomically versioned and reproducible.
A key architectural principle in cloud-native design is: How are Docker images decoupled from environment variables, configuration placeholders, and secrets?
In Pure GitOps, the executable container image and environment-specific configuration are strictly separated at the architectural and lifecycle layer:
- The Docker Image is 100% Environment-Agnostic (12-Factor App):
- The Java 21 / Spring Boot container contains compiled bytecode (
app.jar) and the OpenTelemetry agent. - It contains zero environment endpoints, zero credentials, and zero hardcoded URLs.
- The exact same container image digest (
sha256:abc1234) promoted from DEV runs unchanged in STAGING and PROD.
- The Java 21 / Spring Boot container contains compiled bytecode (
- Environment Variables & Placeholders Live in GitOps Manifests:
- Environment-specific values (
SPRING_PROFILES_ACTIVE, database URLs, Vault secret placeholders) live declaratively in Kustomize overlays (sample-apps/jhipster-microservice/k8s/overlays/{dev,staging,prod}/patch-env.yaml). - At pod startup, Kubernetes/OpenShift injects these values from ConfigMaps and Secrets reconciled by ArgoCD.
- Environment-specific values (
Click to expand: 📦 Docker Image vs. Environment Variables Decoupling Lifecycle Diagram
flowchart LR
subgraph BuildTime["1. CI Build (Image)"]
direction TB
Code["☕ Java 21 Code<br/>(Maven Package)"] --> Build["🏗️ Lean Jenkins CI"]
Build --> Image["🐳 Immutable Image<br/>(sha256:abc1234)<br/>• Zero env endpoints<br/>• Zero secrets"]
Image --> Registry["OpenShift Registry"]
end
subgraph Runtime["2. CD GitOps (Config)"]
direction TB
Overlays["📁 GitOps Manifests<br/>• overlays/dev/env<br/>• overlays/prod/env<br/>• ConfigMaps / Vault"]
ArgoCD["🐙 ArgoCD 3.5 Engine"]
Cluster["☸️ OpenShift Cluster"]
Overlays --> ArgoCD
ArgoCD -->|"Injects at Startup"| Cluster
end
Registry -.->|"Pulls by Digest"| Cluster
Rather than fragmenting into 3 separate Git repositories (app-repo, ci-platform-repo, and global-vars-repo), this project organizes them in a cohesive, self-contained Platform Blueprint:
- 1-Click Reproducibility & Portability:
Platform engineers can clone a single repository and execute
./deploy.shormake deployto provision Jenkins JCasC, ArgoCD 3.5, ApplicationSets, Observability, and sample microservices with zero broken links. - Elimination of Jenkins Cross-Repo Parameter Coordination:
In the legacy push model, separate repositories were needed because Jenkins had a manual UI form requiring developers to pick combinations of
(App_Tag, Vars_Tag). In Pure GitOps, Jenkins does not coordinate parameters—it is event-driven via webhooks. - Atomic Versioning & Traceability:
Every commit represents a verified snapshot where Jenkins pod templates (
jcasc/), pipeline steps (jenkinsfiles/), ArgoCD ApplicationSets (argocd-apps/), and workload overlays (sample-apps/) are guaranteed to be mutually compatible. - Prevention of Orphaned Remote Dependencies: Avoids dependency on external standalone repos that might experience breaking changes, access revocations, or deletion.
| Architectural Dimension | Platform Monorepo (This Blueprint) | Enterprise Polyrepo GitOps |
|---|---|---|
| Repository Topology | 1 Cohesive Repository containing IaC, CI, GitOps manifests, and sample apps. | 2 to 3 Repositories (app-microservice, ci-platform, central-gitops-manifests). |
| Target Audience | Reference architectures, blueprints, platform teams, PoCs, and fast onboarding. | Large enterprises with strict organization boundaries (Dev Teams vs. Platform Ops). |
| Docker Image Decoupling | Fully Decoupled: Image is built environment-agnostic; overlays inject runtime config. | Fully Decoupled: Image is built environment-agnostic; overlays inject runtime config. |
| Jenkins CI Role | Parameterless Multibranch CI: Builds, scans, signs, and commits tag to local GitOps folder. | Parameterless Multibranch CI: Builds, scans, signs, and opens a PR to central GitOps repo. |
| Jenkins UI Parameters | Zero Parameters (Pure webhook / event-driven). | Zero Parameters (Pure webhook / event-driven). |
| ArgoCD GitOps Role | Reconciles manifests directly from sample-apps/gitops-manifests/ or overlays. |
Reconciles manifests directly from the central gitops-manifests repository. |
| Rollback & Auditability | Instant git revert or ArgoCD 1-click revision rollback. |
Instant git revert in the GitOps repo or ArgoCD 1-click revision rollback. |
💡 Architectural Summary & Conclusion: Complete decoupling between application code and environment-specific configuration is achieved at the container boundary rather than through fragmented Git repositories. The exact same immutable, SLSA-signed container image runs across DEV, STAGING, and PROD, while environment variables, secrets, and placeholders are injected dynamically at pod startup via declarative Kustomize overlays.
A critical enterprise architectural consideration is: How do Internal Developer Portals (Backstage IDP) and ITSM Change Management platforms (ServiceNow, Jira Service Management) integrate with this Pure GitOps pattern?
In the legacy push architecture (jenkins-git-parameter Pattern 2), Backstage and ITSM platforms called Jenkins Pipeline 02 via REST API (POST /job/02-CD-Release-Orchestrators/job/multi-cluster-release-orchestrator/buildWithParameters), making Jenkins the central orchestrator and security vulnerability.
In Pure GitOps (jenkins-without-git-parameter), Backstage and ServiceNow interact directly with Git (the Single Source of Truth) and ArgoCD 3.5, while ArgoCD Notifications automatically updates tickets and developer catalogs:
Click to expand: 🔄 Side-by-Side ITSM & Backstage Architectural Flow (Push vs. Pure GitOps)
flowchart TB
subgraph LegacyITSM["Pattern A: Push ITSM"]
direction TB
DevA["👩💻 Developer /<br/>Release Manager"] -->|"1. Opens Ticket"| ITSMA["📋 ServiceNow / Jira<br/>(CHG00123)"]
ITSMA -->|"2. Approved: API"| JMasterA["⚙️ Jenkins Master<br/>(REST API Trigger)"]
JMasterA -->|"3. Runs Pipeline 02"| JAgentA["🚀 Jenkins Agent Pod<br/>• Holds Cluster Keys<br/>• Skopeo Engine"]
JAgentA -->|"4. Commits & Syncs"| ArgoA["🐙 ArgoCD Controller"]
ArgoA -->|"5. Deploys"| K8sA["☸️ OpenShift PROD"]
JAgentA -->|"6. Closes Ticket"| ITSMA
BackstageA["🎭 Backstage IDP"] -->|"Trigger Job"| JMasterA
end
subgraph PureGitOpsITSM["Pattern B: GitOps ITSM"]
direction TB
DevB["👩💻 Developer /<br/>Release Manager"] -->|"1. Self-Service"| PortalB["🎭 Backstage / Jira<br/>(CHG00123)"]
PortalB -->|"2. Merge PR (CHG00123)"| GitB["🐙 GitOps Repo (SSOT)<br/>(Protected branch)"]
GitB -->|"3. Continuous Sync"| ArgoB["🐙 ArgoCD 3.5 Engine<br/>(AppSets & Rollouts)"]
ArgoB -->|"4. Progressive Sync"| K8sB["☸️ OpenShift PROD"]
ArgoB -.->|"5. Notifications: OK"| PortalB
ArgoB -.->|"6. ArgoCD Plugin"| PortalB
end
💡 Architectural Summary & Conclusion: Integrating Backstage IDP and ServiceNow / Jira ITSM with Pure GitOps replaces legacy imperative Jenkins API triggers with declarative GitOps pull requests. This guarantees a cryptographically signed audit trail for SOX/SOC2 compliance, removes cluster deployment tokens from developer portals, and leverages ArgoCD Notifications to automate change ticket closures upon verified cluster health.
Click to expand: ⚡ ServiceNow / Jira ITSM Automated Change Approval & Reconciliation Sequence Diagram
sequenceDiagram
autonumber
actor Ops as 👩💻 Release Manager
participant ITSM as 📋 ServiceNow / Jira
participant GitHub as 🐙 GitHub (GitOps Repo)
participant Jenkins as 🏗️ Lean Jenkins CI
participant ArgoCD as 🐙 ArgoCD 3.5 Engine
participant Cluster as ☸️ OpenShift PROD
Note over Jenkins: 1. CI Build, Syft SBOM,<br/>Trivy Scan & Cosign SLSA 3
Jenkins-->>ITSM: Attach Evidence (SBOM & Cosign)
Ops->>ITSM: Review & Approve CHG009876
ITSM->>GitHub: Merge Promotion PR #89
GitHub->>ArgoCD: Git Push Event on 'prod'
activate ArgoCD
ArgoCD->>Cluster: Progressive Canary (Argo Rollouts)
Cluster-->>ArgoCD: Health Checks (HTTP 200)
deactivate ArgoCD
ArgoCD-->>ITSM: Notifications: Close CHG009876
ArgoCD-->>ITSM: Post Deployment Audit Metrics
- 🎭 Backstage IDP (Developer Self-Service & Catalog):
- Promotion Scaffolder Template: Backstage uses
@backstage/plugin-scaffolder-backendactionpublish:github:pull-requestto create or merge Promotion PRs in the GitOps repository. - Live Cluster Observability: Developers view real-time sync status, health checks, pod logs, and canary rollout graphs directly within the Backstage Entity Page via
@roadiehq/backstage-plugin-argo-cdor Red Hat Developer Hub (RHDH).
- Promotion Scaffolder Template: Backstage uses
- 📋 ServiceNow / Jira ITSM (Change Management & SOX Compliance):
- Automated Evidence Collection: Jenkins CI attaches the Cosign SLSA 3 signature link, CycloneDX SBOM, and Trivy vulnerability scan report directly to the pending ServiceNow Change Request (
CHG009876). - Automated Ticket Closure: The ArgoCD Notifications Controller detects healthy cluster deployment and sends a webhook to ServiceNow (
PATCH /api/now/table/change_request/<id>) transitioning the ticket toClosed / Implemented.
- Automated Evidence Collection: Jenkins CI attaches the Cosign SLSA 3 signature link, CycloneDX SBOM, and Trivy vulnerability scan report directly to the pending ServiceNow Change Request (
| Integration Dimension | Legacy Pattern 2 (jenkins-git-parameter) |
Pure GitOps (jenkins-without-git-parameter) |
Advantage & Recommendation |
|---|---|---|---|
| Trigger Mechanism | Imperative API: Backstage/ITSM calls Jenkins buildWithParameters. |
Declarative Git: Backstage/ITSM creates or merges a GitOps PR. | 🏆 GitOps: Standard Git audit trail (cryptographically signed). |
| Credential Storage | Backstage/ServiceNow stores Jenkins admin credentials; Jenkins holds cluster tokens. | Zero-Trust: ServiceNow only holds a scoped GitHub PR merge token. No cluster tokens. | 🏆 GitOps: Minimal attack surface. |
| Developer Experience in Backstage | Blind: Backstage only shows Jenkins job console logs. | Rich & Live: Backstage ArgoCD Plugin renders live Pod topology, sync status, and rollout progress. | 🏆 GitOps: Superior inner-to-outer loop developer UX. |
| Audit & Compliance (SOX / SOC2) | Fragmented across Jenkins build history and ServiceNow. | Unified in Git: Every production change has an immutable Git commit, peer reviews, and ServiceNow Change ID. | 🏆 GitOps: 100% auditable Single Source of Truth. |
| Failure & Rollback Flow | Developer must log into Jenkins or ServiceNow and manually re-run an older build. | Instant: git revert <commit> in Git, or click 1-click Rollback in ArgoCD/Backstage UI. |
🏆 GitOps: Sub-second deterministic rollback. |
| Post-Deploy Ticket Closure | Jenkins Pipeline script executes custom Groovy REST calls at the end of the job. | Native ArgoCD Notifications: Event-driven webhook engine handles ticket state transitions. | 🏆 GitOps: Resilient; not tied to a running Jenkins pod. |
Click to expand: 🗺️ End-to-End Multi-Cluster Platform Topology Diagram
flowchart TB
subgraph DeveloperWorkspace["1. Dev & Git (SSOT)"]
direction TB
Dev["👩💻 Developer /<br/>Release Manager"]
AppRepo["📦 App Microservice<br/>(sample-apps/jhipster)"]
GitOpsRepo["🌐 GitOps Repository<br/>(Manifests & Config)"]
GitHubPR["🔀 GitHub Pull Requests"]
end
subgraph OCP_DEV["Cluster 1: DEV"]
direction TB
subgraph JenkinsPlatform["Jenkins CI (Lean)"]
Master["Jenkins Controller<br/>(JCasC & Multibranch)"]
Seed["00-Seed-Job<br/>Provisioner"]
CIJob["01-Multibranch-CI<br/>(Webhook Triggered)"]
end
subgraph Agents["Ephemeral Agent Pods"]
MavenAgent["maven-jdk21 Agent<br/>(Build & Test)"]
SecurityAgent["security-tools Agent<br/>(Cosign & Syft)"]
end
subgraph OCPDevRegistry["Internal Registry"]
DevReg["image-registry:5000<br/>nubenetes-dev-apps"]
end
subgraph ArgoCDMaster["ArgoCD 3.5 Engine"]
ArgoServer["ArgoCD Server &<br/>ApplicationSets"]
PRGen["PR Preview Generator"]
MatrixGen["Matrix Generator"]
end
subgraph ObservabilityStack["Observability Stack"]
OTel["OTel Collector<br/>(OTLP Tracing)"]
Prom["Prometheus Server<br/>(Metrics Scraping)"]
Grafana["Grafana 13.2.0<br/>(Dashboards)"]
end
DevApps["DEV Workloads<br/>(dev-apps)"]
PreviewApps["Ephemeral PR Previews<br/>(pr-preview-*)"]
end
subgraph OCP_STG["Cluster 2: STAGING"]
StgApps["Staging Workloads<br/>(staging-apps)"]
end
subgraph OCP_PRD["Cluster 3: PROD"]
PrdApps["Production Workloads<br/>(prod-apps)"]
Rollout["Argo Rollouts<br/>(Canary & SLA)"]
end
%% Developer interactions
Dev -->|"1. Push Code / PR"| AppRepo
Dev -->|"2. Tag / Merge PR"| GitOpsRepo
AppRepo -.->|"Webhook Event"| CIJob
GitHubPR -.->|"PR Webhook"| PRGen
%% Jenkins CI Flow
CIJob -->|"Spawns"| MavenAgent
MavenAgent -->|"Compile & Test"| DevReg
CIJob -->|"Spawns"| SecurityAgent
SecurityAgent -->|"Cosign & Syft"| DevReg
SecurityAgent -->|"3. Commit Digest"| GitOpsRepo
%% ArgoCD GitOps Flow
GitOpsRepo -->|"Continuous Sync"| ArgoServer
ArgoServer -->|"Target: main"| DevApps
PRGen -->|"Target: head_sha"| PreviewApps
MatrixGen -->|"Target: staging"| StgApps
MatrixGen -->|"Target: prod"| Rollout
Rollout -->|"Progressive Canary"| PrdApps
%% Observability
Master -.->|"OTLP Spans"| OTel
ArgoServer -.->|"Sync Metrics"| Prom
OTel -.->|"Traces"| Grafana
Prom -.->|"Metrics Scrape"| Grafana
- Developer & Git Ecosystem (SSOT):
- Application microservice source code and declarative Kubernetes manifests live in a unified, version-controlled repository.
- Developers promote releases exclusively via Git Pull Requests or Semantic Version tags (
v1.2.0).
- Lean Jenkins CI (DEV Cluster):
- Webhook-triggered multibranch pipeline with zero parameter dropdowns.
- Ephemeral
maven-jdk21agent runs unit and integration tests, whilesecurity-toolsagent signs container images with Cosign (SLSA Level 3) and generates Syft SBOMs. - Automatically commits updated image digests to GitOps manifests using scoped bot credentials.
- ArgoCD 3.5 GitOps Engine:
- ApplicationSet controllers continuously pull Git state and reconcile target OpenShift clusters.
- PR Preview Generator creates isolated ephemeral review environments; Matrix Generator synchronizes DEV, STAGING, and PROD.
- Full-Stack Observability:
- OpenTelemetry (OTel) Collector, Prometheus, and Grafana 13.2.0 provide unified trace, metric, and log correlation from CI to production runtime.
💡 Architectural Summary & Conclusion: This multi-cluster topology cleanly separates developer workflows, lean CI automation, and GitOps continuous delivery across OpenShift environments. By delegating all multi-cluster deployment and canary routing to ArgoCD and Argo Rollouts, the platform achieves least-privilege security and sub-second operational responsiveness.
Click to expand: ⚠️ Jenkins SCM Pre-Execution Lifecycle Blindspot Diagram
flowchart TB
subgraph UI_Phase["1. Master (Pre-Exec)"]
direction TB
User["👤 User opens<br/>Build UI Form"]
Master["⚙️ Jenkins Master<br/>reads Job XML"]
GitParam["🔍 git-parameter<br/>queries SCM refs"]
Dropdown["📋 Renders Branch/Tag<br/>Dropdown in Browser"]
User --> Master --> GitParam --> Dropdown
end
subgraph Runtime_Phase["2. Runtime (Agent)"]
direction TB
AllocAgent["☸️ Ephemeral Agent<br/>Pod Allocated"]
RunStage["📦 Pipeline Stage:<br/>checkout repo 2"]
AllocAgent --> RunStage
end
Gap["⚠️ SCM Blindspot:<br/>Dynamic checkouts<br/>occur during build and<br/>invisible at render"]
Dropdown -.-> Gap
Gap -.-> RunStage
- The Pre-Execution Paradox:
- Jenkins calculates and renders job parameters in the browser before allocating an agent pod and before executing any
Jenkinsfilestage. - The
git-parameterplugin can only discover repositories statically configured in the Jenkins Master Job XML.
- Jenkins calculates and renders job parameters in the browser before allocating an agent pod and before executing any
- The SCM Blindspot:
- Any secondary repository cloned dynamically inside a pipeline stage (
checkout repo 2) is completely invisible at UI render time.
- Any secondary repository cloned dynamically inside a pipeline stage (
- The GitOps Solution:
- In Pure GitOps, this entire failure mode is eliminated because Jenkins runs parameterless via webhooks, leaving release selection to native Git and ArgoCD.
💡 Architectural Summary & Conclusion: The Jenkins SCM pre-execution paradox makes multi-repository parameterization fragile in Jenkins declarative pipelines. Pure GitOps sidesteps this engine limitation completely by making Jenkins CI parameterless and event-driven via webhooks, delegating all environment targeting to declarative Git branches and ArgoCD revisions.
Click to expand: 🔄 Side-by-Side Architectural Flow (Push vs. Pull)
flowchart LR
subgraph PatternA["Pattern A: Push Model"]
direction TB
A1["👩💻 User Opens<br/>Jenkins UI"] --> A2["📋 Selects Branch/Tag<br/>in Dropdown"]
A2 --> A3["⚙️ Jenkins Master<br/>Queries Git SCM"]
A3 --> A4["📦 Jenkins Agent<br/>Builds Image"]
A4 --> A5["🚀 Agent Pushes<br/>Direct to Cluster<br/>(oc apply / sync)"]
A5 --> A6["⚠️ Jenkins holds<br/>secrets in CI & drift"]
end
subgraph PatternB["Pattern B: Pull GitOps"]
direction TB
B1["👩💻 Developer Pushes<br/>to Git / Opens PR"] --> B2["⚡ Webhook Triggers<br/>Jenkins CI"]
B2 --> B3["📦 Jenkins Builds &<br/>Signs Image (SLSA 3)"]
B3 --> B4["📝 Auto-Commits Tag<br/>to GitOps Repo"]
B4 --> B5["🔄 ArgoCD Pulls &<br/>Reconciles Cluster"]
B5 --> B6["✅ Result: Zero secrets<br/>in CI & self-healing"]
end
- Pattern A: Jenkins Push Model (Legacy):
- User opens Jenkins UI ➔ selects branches/tags in dropdown ➔ Jenkins agent builds artifact ➔ Jenkins agent holds cluster admin secrets and pushes directly to OpenShift (
oc apply). - Disadvantages: High credential exposure, no continuous drift detection, manual deployment bottlenecks.
- User opens Jenkins UI ➔ selects branches/tags in dropdown ➔ Jenkins agent builds artifact ➔ Jenkins agent holds cluster admin secrets and pushes directly to OpenShift (
- Pattern B: Pure GitOps Pull Model (Recommended):
- Developer pushes to Git ➔ Webhook triggers parameterless Jenkins CI ➔ Jenkins builds, signs (SLSA 3), and auto-commits image digest to Git ➔ ArgoCD pulls and reconciles cluster state.
- Advantages: Zero cluster credentials in CI, continuous 24/7 self-healing, deterministic rollback via
git revert.
💡 Architectural Summary & Conclusion: Moving from imperative Push to declarative Pull eliminates the highest-risk attack vector in traditional CI/CD: storing cluster-admin tokens inside build servers. With ArgoCD pulling state from Git, OpenShift clusters remain continuously aligned with the desired configuration, instantly self-healing from configuration drift.
Click to expand: ⚡ Dynamic Ephemeral PR Preview Environments Sequence Diagram
sequenceDiagram
autonumber
actor Dev as 👩💻 Developer
participant GitHub as 🐙 GitHub (sample-apps)
participant Jenkins as 🏗️ Lean Jenkins CI
participant Registry as 🐳 OpenShift Registry
participant ArgoCD as 🐙 ArgoCD AppSets
participant Cluster as ☸️ OpenShift DEV
Dev->>GitHub: Open PR #42 (feature-branch)
GitHub->>Jenkins: Webhook: PR #42 created
Jenkins->>Jenkins: Build, Trivy scan & Cosign sign
Jenkins->>Registry: Push image: app:pr-42-sha7
GitHub->>ArgoCD: Polling / PR Webhook
Note over ArgoCD: Discovers PR #42<br/>label: preview-env
ArgoCD->>Cluster: Create ns 'pr-preview-42'
ArgoCD->>Cluster: Deploy app (head_sha)
ArgoCD-->>GitHub: Post Preview URL in PR #42
Dev->>Cluster: Verify on live preview ns
Dev->>GitHub: Merge PR #42 into main
GitHub->>ArgoCD: PR closed event
ArgoCD->>Cluster: Tear down ns 'pr-preview-42'
- 1. PR Creation: Developer opens a feature Pull Request (e.g. PR #42) on GitHub with label
preview-environment. - 2. Automated CI Build: Jenkins Multibranch CI compiles Java 21 code, executes Trivy security scans, signs with Cosign, and pushes image
app:pr-42-sha7. - 3. Ephemeral Namespace Provisioning: ArgoCD ApplicationSet PR Generator detects PR #42, provisions isolated namespace
pr-preview-42, and deploys the workload targetinghead_sha. - 4. Developer Feedback: ArgoCD posts the live preview URL directly as a comment on GitHub PR #42 for stakeholder testing.
- 5. Automated Teardown: Merging or closing PR #42 triggers ArgoCD to cleanly destroy the
pr-preview-42namespace and all associated resources.
💡 Architectural Summary & Conclusion: Automated ephemeral preview environments dramatically accelerate developer feedback loops by spinning up isolated, production-like namespaces on demand for every pull request. Automatic resource cleanup upon PR merge prevents cloud cost sprawl while ensuring zero manual intervention.
Click to expand: 🚀 Automated CI -> GitOps Promotion Sequence Diagram
sequenceDiagram
autonumber
participant GitApp as 📦 App Repo (Code)
participant Jenkins as 🏗️ Lean Jenkins CI
participant Registry as 🐳 OpenShift Registry
participant GitOps as 🌐 GitOps Repo (Manifests)
participant ArgoCD as 🐙 ArgoCD 3.5 Engine
participant OCP as ☸️ OpenShift (DEV/STG/PROD)
GitApp->>Jenkins: Git Push to 'main'
activate Jenkins
Jenkins->>Jenkins: Checkout & compile Java 21
Jenkins->>Jenkins: Unit & Integration Tests
Jenkins->>Jenkins: Syft SBOM & Trivy Scan
Jenkins->>Registry: Push image (sha256:abc1234)
Jenkins->>Registry: Sign with Cosign (SLSA 3)
Jenkins->>GitOps: gitopsCommit(app, tag, env)
Note over Jenkins,GitOps: Updates kustomization.yaml<br/>with Bot identity
deactivate Jenkins
GitOps->>ArgoCD: Webhook / Sync Polling
activate ArgoCD
ArgoCD->>ArgoCD: Detect Out-of-Sync Manifest Diff
ArgoCD->>OCP: Reconcile & Deploy to DEV
ArgoCD->>OCP: Verify Health (HTTP 200)
deactivate ArgoCD
- 1. Git Trigger: Developer merges approved PR into
mainbranch, triggering Jenkins webhook. - 2. Compilation & Testing: Jenkins agent compiles Java 21 / Spring Boot 3 code and runs unit & integration tests.
- 3. Supply Chain Hardening: Generates CycloneDX SBOM (Syft), executes vulnerability scanning (Trivy), and signs container image with Cosign (SLSA Level 3).
- 4. GitOps Auto-Commit: Jenkins CI updates
kustomization.yamlwith the new immutable image digest (sha256:...) and commits using a dedicated Bot identity. - 5. Declarative Reconciliation: ArgoCD detects the manifest commit, rolls out the deployment to DEV cluster, and validates HTTP 200 health check.
💡 Architectural Summary & Conclusion: The automated promotion sequence enforces strict software supply chain security (SLSA Level 3, Syft SBOM, Trivy CVE scanning) before any artifact touches cluster manifests. Automated bot commits ensure that release promotion is deterministic, auditable, and hands-free.
Click to expand: ⚙️ ArgoCD Multi-Cluster Matrix Reconciliation Engine Diagram
flowchart TB
subgraph GitOpsSource["1. GitOps Repo (SSOT)"]
direction TB
Manifests["📁 Workload Overlays<br/>• k8s/overlays/dev<br/>• k8s/overlays/stg<br/>• k8s/overlays/prod"]
ClusterList["📋 Cluster Inventory<br/>• config/clusters.yaml<br/>(dev, staging, prod)"]
end
subgraph AppSetEngine["2. AppSet Engine"]
direction TB
Matrix["⚙️ Matrix Engine:<br/>Combines Clusters<br/>x Overlays"]
AppDev["Application: dev<br/>• target: main<br/>• ns: dev-apps"]
AppStg["Application: staging<br/>• target: staging<br/>• ns: staging-apps"]
AppPrd["Application: prod<br/>• target: prod<br/>• ns: prod-apps"]
end
subgraph TargetClusters["3. OpenShift Runtime"]
direction TB
OCPDev["☸️ OCP DEV Cluster<br/>(dev-apps)"]
OCPStg["☸️ OCP STAGING Cluster<br/>(staging-apps)"]
OCPPrd["☸️ OCP PROD Cluster<br/>(prod-apps)"]
end
Manifests --> Matrix
ClusterList --> Matrix
Matrix --> AppDev -->|"Automated Sync"| OCPDev
Matrix --> AppStg -->|"Automated Sync"| OCPStg
Matrix --> AppPrd -->|"Canary Sync Waves"| OCPPrd
-
Input 1 (Workload Overlays): Kustomize overlay manifests in
k8s/overlays/{dev,staging,prod}containing environment-specific replica counts, ConfigMaps, and Vault placeholders. -
Input 2 (Cluster Inventory): Declarative multi-cluster inventory in
config/clusters.yamldefining DEV, STAGING, and PROD API endpoints. -
Matrix Application Generator: Combines
$N$ overlays with$M$ clusters to dynamically generate and manage:-
jhipster-dev: Tracks branchmain, deploys to namespacedev-apps. -
jhipster-staging: Tracks branchstaging, deploys to namespacestaging-apps. -
jhipster-prod: Tracks branchprod, deploys with Argo Rollouts canary sync waves to namespaceprod-apps.
-
💡 Architectural Summary & Conclusion: The ApplicationSet Matrix Generator provides effortless multi-cluster scaling by dynamically pairing cluster inventories with environment overlays. Platform teams can add new target clusters or microservices simply by updating a YAML list, with zero custom pipeline scripting required.
Click to expand: 🛡️ Zero-Trust Security & RBAC Boundary Architecture Diagram
flowchart TB
subgraph UntrustedZone["1. CI Zone"]
direction TB
JenkinsMaster["Jenkins Controller<br/>• No Cluster RBAC"]
JenkinsAgent["Ephemeral Agent Pod<br/>• maven / security"]
Registry["Internal Registry<br/>• Push & Cosign Sig"]
end
subgraph GitOpsTrustZone["2. GitOps Plane"]
direction TB
GitRepo["Git Repository<br/>• Source of Truth"]
ArgoCD["ArgoCD 3.5 Engine<br/>• AppSets Engine"]
end
subgraph WorkloadClusters["3. OpenShift Clusters"]
direction TB
DEV["OCP DEV Cluster<br/>• SCC restricted-v2"]
STG["OCP STAGING Cluster<br/>• SCC restricted-v2"]
PRD["OCP PROD Cluster<br/>• Canary & Hardened"]
end
JenkinsMaster -->|"Spawns"| JenkinsAgent
JenkinsAgent -->|"Push Image & Sig"| Registry
JenkinsAgent -->|"Commit Digest"| GitRepo
JenkinsAgent -.->|"⛔ BLOCKED"| DEV
JenkinsAgent -.->|"⛔ BLOCKED"| STG
JenkinsAgent -.->|"⛔ BLOCKED"| PRD
GitRepo -->|"Continuous Sync"| ArgoCD
ArgoCD -->|"Reconcile via Token"| DEV
ArgoCD -->|"Reconcile via Token"| STG
ArgoCD -->|"Reconcile via Token"| PRD
- Zone 1: CI Workload Zone (Least Privilege):
- Jenkins Controller and build agent pods operate with restricted permissions.
- Strictly Blocked: Jenkins has zero cluster-admin credentials and cannot deploy directly to any OpenShift cluster.
- Zone 2: GitOps Control Plane (High Privilege):
- Only ArgoCD holds short-lived service account tokens to reconcile desired Kubernetes state.
- All deployment actions are triggered solely by verified, cryptographically signed Git commits.
- Zone 3: Protected Workload Runtime:
- Workload pods run under OpenShift Security Context Constraints (
SCC restricted-v2), non-root users, and read-only root filesystems.
- Workload pods run under OpenShift Security Context Constraints (
💡 Architectural Summary & Conclusion: The zero-trust boundary strictly isolates untrusted CI workloads from high-privilege deployment control planes. Jenkins agents have zero network access or RBAC permissions to deploy to Kubernetes, preventing container breakout attacks from compromising multi-cluster infrastructure.
Click to expand: 📊 Progressive Delivery & Prometheus Metric Analysis Diagram
flowchart TB
subgraph RolloutController["1. Argo Rollouts"]
direction TB
Step1["1. Initiate Canary<br/>(Set Weight to 20%)"]
Step2["2. Prometheus Analysis<br/>• Error Rate < 0.5%<br/>• Latency p95 < 200ms"]
Step3["3. Promote Canary<br/>(Set Weight to 50%)"]
Step4["4. Full Production<br/>Promotion (100%)"]
Abort["🚨 Auto-Rollback<br/>to Stable Version"]
end
subgraph RoutingLayer["2. Ingress & Routing"]
direction TB
CanaryService["Canary Service Pods<br/>(20% Test Traffic)"]
StableService["Stable Service Pods<br/>(80% Live Traffic)"]
end
Step1 --> CanaryService
Step1 --> StableService
CanaryService -->|"Telemetry Spans"| Step2
Step2 -->|"SLA Passed"| Step3
Step2 -->|"SLA Breached"| Abort
Step3 --> Step4
- 1. Initial Canary Split: Argo Rollouts routes 20% of incoming live user traffic to new canary pods and 80% to stable pods.
-
2. Automated Prometheus SLA Analysis: Background
AnalysisRunqueries Prometheus for 5 minutes evaluating:- HTTP error rate
$< 0.5%$ . - p95 request latency
$< 200 ext{ms}$ .
- HTTP error rate
- 3. Progressive Traffic Scaling: If metrics pass, traffic scales to 50% and subsequently to 100% full production rollout.
- 4. Instant Automated Rollback: If SLAs breach at any step, traffic is immediately redirected to stable pods and the canary is aborted.
💡 Architectural Summary & Conclusion: Progressive delivery with Argo Rollouts and Prometheus metrics replaces all-or-nothing deployments with automated, data-driven canary promotions. Automated SLA validation ensures that faulty releases are instantly aborted before impacting the majority of end users.
Click to expand: 🔭 Full-Stack Observability & W3C Trace Context Propagation Diagram
flowchart LR
subgraph PipelineSpan["1. Jenkins CI Span"]
direction TB
JTrace["OTel CI Visibility<br/>Trace: 7492...048"]
JBuild["mvn clean package"]
JSign["cosign sign & syft"]
end
subgraph GitOpsSpan["2. GitOps Span"]
direction TB
GCommit["git commit (trace)"]
ASync["ArgoCD Sync Check"]
end
subgraph AppRuntimeSpan["3. Java Runtime Span"]
direction TB
AppStart["JVM App Startup"]
HTTPReq["REST /api/orders"]
end
subgraph UnifiedGrafana["4. Grafana 13.2.0"]
GrafanaDash["Grafana APM Dashboard<br/>Traces & Metrics"]
end
JTrace --> JBuild --> JSign
JSign --> GCommit --> ASync
ASync --> AppStart --> HTTPReq
PipelineSpan -.->|"OTLP Exporter"| UnifiedGrafana
GitOpsSpan -.->|"Prometheus Sync"| UnifiedGrafana
AppRuntimeSpan -.->|"OTel Agent"| UnifiedGrafana
- 1. Jenkins CI Span: OpenTelemetry Jenkins plugin starts trace
7492...048capturing build duration, Maven test results, and Cosign signing steps. - 2. GitOps & ArgoCD Span: Trace ID is injected into Git commit metadata and tracked across ArgoCD sync and health check events.
- 3. Java Application Runtime Span: Spring Boot microservice receives incoming requests and propagates W3C
traceparentheaders via OpenTelemetry Java Agent. - 4. Unified Grafana 13.2.0: SREs and developers correlate pipeline performance, GitOps deployment events, and live runtime APM traces in a single dashboard.
💡 Architectural Summary & Conclusion: Propagating W3C trace context from Jenkins build steps through Git commits and into runtime application requests bridges the visibility gap between CI/CD and production APM. SRE teams can trace latency anomalies or production regressions directly back to the exact commit and pipeline build that introduced them.
jenkins-git-parameter: Requires managing legacy Jenkins plugins (git-parameter,active-choices,job-dslmulti-remote SCM hacks). As the number of microservices grows to dozens or hundreds, maintaining Jenkins Job XML definitions and handling SCM caching issues becomes an operational nightmare.jenkins-without-git-parameter(Pure GitOps): Jenkins requires zero custom parameter plugins. It operates as a standard multibranch CI engine triggered by git events. The developer interface is Git itself (pull requests, release tags), which all developers already know. Pure GitOps is far easier to operate, automate, and scale.
jenkins-git-parameter: Jenkins agents must be given direct deployment credentials (Kubernetes service account tokens with create/update rights, ArgoCD admin tokens). A compromised Jenkins build script can compromise the entire OpenShift cluster.jenkins-without-git-parameter(Pure GitOps): Jenkins agents only have write access to the container registry and Git repository. Jenkins has zero cluster credentials. Even if a build container is compromised, the attacker cannot modify the production cluster.
jenkins-git-parameter: Deployments are imperative events. If a cluster goes down, re-creating the exact state of all applications requires identifying and re-running specific historical Jenkins builds. Configuration drift is invisible to Jenkins.jenkins-without-git-parameter(Pure GitOps): Git holds the complete, declarative desired state. If a cluster is destroyed, pointing ArgoCD at the GitOps repository restores 100% of all microservices, configs, routes, and policies in minutes. Rollback is as simple asgit revert.
jenkins-git-parameter: Forces developers to leave their IDE and Git workflow, navigate to Jenkins UI, wait for dropdowns to populate from remote Git, fill out forms, and trigger builds manually.jenkins-without-git-parameter(Pure GitOps): Developers remain entirely in Git. Opening a PR automatically triggers CI tests and spins up an ephemeral preview environment via ArgoCD ApplicationSets. Merging tomaindeploys to DEV. Taggingv1.0.0deploys to PROD.
Tip
The Pure GitOps pattern is strongly recommended for all enterprise cloud-native workloads.
- Easier to Maintain: No Jenkins SCM multi-remote hacks, no parameter plugin bugs, no pre-execution lifecycle bottlenecks.
- Significantly More Secure: Zero-trust architecture where CI has no cluster deployment tokens.
- Standardized & Future-Proof: Aligned with CNCF, OpenGitOps, and Red Hat OpenShift reference architectures.
The Jenkins Controller is deployed using the official Jenkins Helm chart configured via Jenkins Configuration as Code (JCasC).
- Plugins (
helm/jenkins/plugins.txt): Contains strictly core multibranch, kubernetes, opentelemetry, and security plugins.git-parameteris completely excluded. - JCasC (
jcasc/jenkins-jcasc.yaml): Provisions security matrix, OpenTelemetry exporter endpoints, and seed job automation. - Job DSL (
jobdsl/pipelines-ci.groovy): DeclaresmultibranchPipelineJobinstances that automatically scan branches and pull requests. - Ephemeral Agents (
jcasc/pod-templates.yaml): Kubernetes pod templates formaven-jdk21,node-angular,security-tools, andgitops-botcomplying with OpenShiftrestricted-v2SCC.
ArgoCD 3.5 serves as the single source of truth for continuous deployment across clusters:
- Root App-of-Apps (
argocd-apps/root-app-of-apps.yaml): Bootstraps all platform applications declaratively. - Matrix Generator (
argocd-apps/applicationset-clusters.yaml): Uses ApplicationSet Matrix Generators to deploy workloads across all registered OpenShift clusters (DEV,STAGING,PROD). - PR Preview Generator (
argocd-apps/applicationset-pull-request-preview.yaml): Dynamically creates and destroys preview environments per GitHub PR. - TargetRevision Management:
- DEV:
targetRevision: main(Continuous deployment on merge). - STAGING:
targetRevision: staging(or semver release tags). - PROD:
targetRevision: prod(Protected release branch with Argo Rollouts Canary).
- DEV:
The CI pipeline implements end-to-end software supply chain security:
- Vulnerability Scanning: Aqua Security Trivy scans all application binaries and container base layers.
- SBOM Generation: Anchore Syft generates CycloneDX Software Bill of Materials.
- Container Signing & Attestation: Sigstore Cosign signs the container image digest and attaches the SBOM attestation.
- OpenShift Image Signature Policy (
security/openshift-image-signature-policy.yaml): Enforces that only Cosign-signed images can run in production.
- OpenTelemetry Plugin: Emits W3C distributed trace spans (
traceparent) from Jenkins CI stages to the OTel Collector. - ArgoCD Metrics: Exported directly to Prometheus for sync duration, health status, and drift alerts.
- Grafana 13.2.0 Dashboards (
observability/dashboards/):argocd-gitops-sync.json: Real-time GitOps synchronization and cluster drift monitoring.jenkins-performance-otel.json: CI build duration, queue latency, and pipeline execution traces.app-full-stack-correlation.json: End-to-end correlation from Git commit -> Jenkins CI build -> ArgoCD sync -> Spring Boot microservice runtime.
.
├── .gitignore
├── LICENSE
├── Makefile # Automation CLI (deploy, destroy, reinstall, promote)
├── README.md # Master architectural blueprint & documentation
├── deploy.sh # 1-Click platform deployment script
├── destroy.sh # Clean teardown script
├── reinstall.sh # Full wipe and fresh redeployment
├── config/
│ ├── clusters.yaml # Multi-cluster topologies (DEV, STG, PRD)
│ └── environments.env # Environment variables & platform domain endpoints
├── argocd-apps/
│ ├── root-app-of-apps.yaml # ArgoCD Root App-of-Apps bootstrapper
│ ├── applicationset-clusters.yaml # Multi-cluster matrix generator
│ ├── applicationset-pull-request-preview.yaml# Dynamic PR preview environment generator
│ ├── applicationset-git-branches.yaml # Git branch/tag tracking generator
│ └── apps/
│ ├── dev/
│ │ ├── jhipster-microservice.yaml # DEV Application (targetRevision: main)
│ │ └── grafana-observability.yaml # Observability stack application
│ ├── staging/
│ │ └── jhipster-microservice.yaml # STAGING Application (targetRevision: staging)
│ └── prod/
│ └── jhipster-microservice.yaml # PROD Application (targetRevision: prod / Canary)
├── helm/
│ ├── jenkins/
│ │ ├── Chart.yaml
│ │ ├── values.yaml # Standard Jenkins Helm values
│ │ ├── values-openshift.yaml # OpenShift 4.20+ hardened values (restricted-v2)
│ │ └── plugins.txt # Lean plugins (NO git-parameter plugin)
│ ├── argocd/
│ │ └── values-argocd-3.5.yaml # ArgoCD 3.5 Helm values with ApplicationSets
│ └── observability/
│ ├── grafana-values.yaml # Grafana 13.2.0 with OTel & Prometheus datasources
│ ├── prometheus-values.yaml # Prometheus metrics configuration
│ └── otel-collector-values.yaml # OpenTelemetry OTLP Collector configuration
├── jcasc/
│ ├── jenkins-jcasc.yaml # JCasC Master configuration
│ ├── pod-templates.yaml # Kubernetes agent pod templates
│ └── github-app-credentials.yaml # GitHub App credentials for SCM & GitOps PRs
├── jenkinsfiles/
│ └── ci/
│ ├── Jenkinsfile.app-java-maven # Pure CI: Test, build, Trivy, Syft, Cosign, GitOps commit
│ └── Jenkinsfile.app-angular # Pure CI for Angular frontend
├── jobdsl/
│ ├── seed-job.groovy # Master seed job folder provisioner
│ └── pipelines-ci.groovy # Multibranch Pipeline Job DSL (Zero parameters)
├── observability/
│ └── dashboards/
│ ├── argocd-gitops-sync.json # ArgoCD GitOps sync dashboard
│ ├── jenkins-performance-otel.json # Jenkins pipeline traces & metrics dashboard
│ └── app-full-stack-correlation.json # End-to-end trace correlation dashboard
├── sample-apps/
│ ├── jhipster-microservice/ # Java 21 / Spring Boot 3 cloud-native microservice
│ │ ├── Dockerfile
│ │ ├── pom.xml
│ │ ├── k8s/ # Base & overlays for dev, staging, prod
│ │ ├── rollout/ # Argo Rollouts Canary & AnalysisTemplate
│ │ └── src/...
│ └── gitops-manifests/ # Dedicated GitOps configuration repository
│ ├── apps/applications-inventory.yaml
│ ├── clusters/
│ └── environments/
├── scripts/
│ ├── deploy.sh
│ ├── destroy.sh
│ ├── generate-tokens.sh
│ ├── gitops-promote.sh # CLI helper for GitOps release promotion
│ ├── ocp-setup-scc.sh # OpenShift SCC restricted-v2 configuration
│ └── setup-argocd-clusters.sh # Multi-cluster ArgoCD secret registration
├── security/
│ ├── openshift-image-signature-policy.yaml # Cosign image signature verification policy
│ └── external-secrets-operator/ # Vault / ESO zero-trust secrets integration
└── shared-library/
├── src/com/nubenetes/gitops/
└── vars/
├── cosignSign.groovy # Cosign container signing step
├── sbomGenerate.groovy # Syft SBOM generation step
├── gitopsCommit.groovy # Automated GitOps commit / PR step
├── otelLogEvent.groovy # OTel span event logger
└── skopeoPromote.groovy # Skopeo container image promotion step
- Red Hat OpenShift Container Platform (OCP) 4.20+ or Kubernetes 1.31+ cluster.
ocorkubectlCLI logged in withcluster-adminpermissions.helmv3.14+ installed.
Clone the repository and run the automated deployment script:
git clone https://github.com/nubenetes/jenkins-without-git-parameter.git
cd jenkins-without-git-parameter
# Deploy the entire platform (Jenkins, ArgoCD 3.5, ApplicationSets, Observability)
./deploy.sh
# or
make deployOnce deployed, access the web consoles via your OpenShift router domain:
- Jenkins CI Controller:
https://jenkins-jenkins.apps.ocp-dev.nubenetes.internal - ArgoCD 3.5 Control Plane:
https://argocd-server.apps.ocp-dev.nubenetes.internal - Grafana 13.2.0 Dashboards:
https://grafana.apps.ocp-dev.nubenetes.internal
- Developer creates branch
feature/auth-redesignand opens a PR in GitHub. - Jenkins Multibranch CI automatically compiles, tests, and signs the image.
- ArgoCD ApplicationSet PR Generator automatically provisions namespace
pr-preview-<number>on OpenShift. - When the PR is merged, the preview environment is automatically destroyed.
- Merging PR to
mainbranch triggers Jenkins CI. - CI builds
jhipster-microservice:dev-<sha7>and updatessample-apps/gitops-manifests/environments/dev.yaml. - ArgoCD detects the Git commit and reconciles
jhipster-microservice-devonocp-dev.
Use the GitOps promotion helper script to update the desired environment:
# Promote release to STAGING
./scripts/gitops-promote.sh jhipster-microservice staging v1.2.0
# Promote release to PROD (Triggers Argo Rollouts Canary)
./scripts/gitops-promote.sh jhipster-microservice prod v1.2.0Commit and push the change to Git. ArgoCD immediately detects the change and synchronizes the target cluster with zero Jenkins manual intervention!
To tear down all deployed workloads, namespaces, and Helm releases:
./destroy.sh
# or
make destroyTo execute a complete clean wipe and fresh redeployment:
./reinstall.sh
# or
make reinstall- GitOps & Cloud-Native Standards:
- OpenGitOps Principles & Standards: https://opengitops.net/
- CNCF GitOps Working Group: https://github.com/open-gitops/documents
- ArgoCD & ApplicationSets:
- ArgoCD 3.5 Official Documentation: https://argo-cd.readthedocs.io/en/stable/
- ArgoCD ApplicationSet Generators (PR & Matrix): https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-List/
- Argo Rollouts Progressive Delivery & Canary Analysis: https://argoproj.github.io/argo-rollouts/
- Jenkins Core & Multibranch Pipelines:
- Jenkins Configuration as Code (JCasC): https://www.jenkins.io/projects/jcasc/
- Jenkins Multibranch Pipeline Specification: https://www.jenkins.io/doc/book/pipeline/multibranch/
- Jenkins GitHub Branch Source Plugin: https://plugins.jenkins.io/github-branch-source/
- Supply Chain Security & Attestation (SLSA Level 3):
- Sigstore Cosign Container Signing: https://docs.sigstore.dev/cosign/overview/
- SLSA (Supply-chain Levels for Software Artifacts) Specification: https://slsa.dev/
- Anchore Syft SBOM Generator: https://github.com/anchore/syft
- Aqua Security Trivy Vulnerability Scanner: https://trivy.dev/
- Red Hat OpenShift 4.20+ Hardening:
- OpenShift Container Platform Security & SCCs: https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/authentication_and_authorization/managing-pod-security-policies
- OpenShift Image Signature Enforcement: https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/images/image-signatures
- Full-Stack Observability:
- OpenTelemetry (OTel) Project: https://opentelemetry.io/
- Grafana 13.2.0 Dashboards & Traces: https://grafana.com/docs/grafana/latest/
- Prometheus Monitoring System: https://prometheus.io/