Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
910c489
add schemathesis
AliakseiChahan Jun 4, 2026
558b643
Merge branch 'qa/issue-182' into 'master'
Jun 11, 2026
d4bdfa2
#312-Indexer feature completeness (enable all features and full test …
d0wnvk Jun 15, 2026
a2599af
Merge branch '312-Indexer-feature-completeness' into 'master'
d0wnvk Jun 15, 2026
5a1d850
Migrate to os-core-common 7.1.0
shjellvik Jun 19, 2026
668f08d
Merge branch 'feature/migrate-to-latest-os-core-common' into 'master'
shjellvik Jun 19, 2026
76134f7
[GONRG-14366] GC: add probes for service
Jun 23, 2026
ea12177
Merge branch 'gc-add-probes' into 'master'
Jun 23, 2026
3bd015b
Fix azure_test pipeline step
jorchiu Jun 23, 2026
32bb393
Merge branch 'jordanchiu/fix_azure_test' into 'master'
jorchiu Jun 23, 2026
3039990
Fix to address DefaultAzureCredential in partitionservice
Jun 23, 2026
ee90901
Merge branch 'santoshsinha/defaultazurecedential' into 'master'
jorchiu Jun 23, 2026
f24182d
Align Istio JWT enforcement and clarify mTLS behavior
siarhei-s-epam Jun 24, 2026
a55551b
Merge branch 'namespace-requestAuthentication-policy' into 'master'
siarhei-s-epam Jun 24, 2026
7a02d21
Acceptance test hardening
Jun 25, 2026
74368ba
Merge branch 'acceptance-test-hardening' into 'master'
Jun 25, 2026
28afaf3
fix(azure): netty-bom before core-lib-azure (lettuce 7.5.2 NoClassDef…
shjellvik Jul 1, 2026
6bb0fb3
Merge branch 'fix/azure-netty-bom-lettuce' into 'master'
shjellvik Jul 1, 2026
74946fc
chore(deps): remediate partition vulnerabilities
Jul 15, 2026
2efe9da
Merge branch 'fix/issue-189-dependency-vulnerabilities' into 'master'
Jul 15, 2026
f9d2119
chore(sonar): remove unused imports in partition-aws AuthorizationSer…
marcburnie Jul 16, 2026
82b0fbe
Merge branch 'fix/sonar-s1128-partition-aws-authorizationservice' int…
marcburnie Jul 16, 2026
e4a2a59
Merge remote-tracking branch 'upstream/master' into sync/upstream-202…
github-actions[bot] Jul 18, 2026
b7fedfe
feat: sync upstream changes from
github-actions[bot] Jul 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitlab-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ variables:

ACCEPTANCE_TEST_DIR: "partition-acceptance-test"

CIMPL_OPENAPI_YAML_ROUTE: /api/partition/v1/api-docs.yaml

include:
- project: "osdu/platform/ci-cd-pipelines"
file: "standard-setup.yml"
Expand Down Expand Up @@ -60,6 +62,9 @@ include:
- project: "osdu/platform/ci-cd-pipelines"
file: "cloud-providers/gc-one-pipeline.yml"

- project: "osdu/platform/ci-cd-pipelines"
file: "reporting/schemathesis.yml"

- local: "devops/gc-quarkus/pipeline/override-stages.yml"

- local: "devops/core-plus/pipeline/override-stages.yml"
Expand All @@ -80,3 +85,6 @@ download_plugins:
rules:
- if: "$PROTECTED == '1'"
when: on_success

schemathesis-tests:
extends: .schemathesis-tests-template
14 changes: 13 additions & 1 deletion NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -162,9 +162,13 @@ The following software have components provided under the terms of this license:
- Netty Reactive Streams Implementation (from https://repo1.maven.org/maven2/com/typesafe/netty/netty-reactive-streams)
- Netty/Buffer (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-buffer)
- Netty/Codec (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-codec)
- Netty/Codec/Base (from https://netty.io/)
- Netty/Codec/Compression (from https://netty.io/)
- Netty/Codec/DNS (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-codec-dns)
- Netty/Codec/HTTP (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-codec-http)
- Netty/Codec/HTTP2 (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-codec-http2)
- Netty/Codec/Marshalling (from https://netty.io/)
- Netty/Codec/Protobuf (from https://netty.io/)
- Netty/Codec/Socks (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-codec-socks)
- Netty/Common (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-common)
- Netty/Handler (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-handler)
Expand Down Expand Up @@ -288,6 +292,7 @@ The following software have components provided under the terms of this license:

- Apache Log4j Core (from https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core)
- HdrHistogram (from http://hdrhistogram.github.io/HdrHistogram/)
- Jackson-core (from http://wiki.fasterxml.com/JacksonHome, https://github.com/FasterXML/jackson-core)
- Jodd Util (from https://util.jodd.org/)
- LatencyUtils (from http://latencyutils.github.io/LatencyUtils/)
- MinLog (from https://github.com/EsotericSoftware/minlog)
Expand Down Expand Up @@ -321,6 +326,13 @@ The following software have components provided under the terms of this license:
- ReflectASM (from https://github.com/EsotericSoftware/reflectasm)
- Spring Core (from http://www.springframework.org, https://github.com/spring-projects/spring-framework, https://repo1.maven.org/maven2/org/springframework/spring-core)

========================================================================
BSL-1.0
========================================================================
The following software have components provided under the terms of this license:

- Jackson-core (from http://wiki.fasterxml.com/JacksonHome, https://github.com/FasterXML/jackson-core)

========================================================================
BouncyCastle
========================================================================
Expand Down Expand Up @@ -408,7 +420,6 @@ The following software have components provided under the terms of this license:
- JUnit Platform Engine API (from http://junit.org/junit5/, https://junit.org/junit5/)
- Jakarta Activation API (from https://github.com/eclipse-ee4j/jaf, https://github.com/jakartaee/jaf-api, https://repo1.maven.org/maven2/jakarta/activation/jakarta.activation-api)
- Jakarta Annotations API (from https://projects.eclipse.org/projects/ee4j.ca)
- Jakarta Dependency Injection (from https://github.com/eclipse-ee4j/injection-api)
- Jakarta Messaging API (from https://projects.eclipse.org/projects/ee4j.jms)
- Jakarta XML Binding API (from https://repo1.maven.org/maven2/jakarta/xml/bind/jakarta.xml.bind-api, https://repo1.maven.org/maven2/org/jboss/spec/javax/xml/bind/jboss-jaxb-api_2.3_spec)
- Old JAXB Core (from <https://eclipse-ee4j.github.io/jaxb-ri/>, https://eclipse-ee4j.github.io/jaxb-ri/, https://repo1.maven.org/maven2/com/sun/xml/bind/jaxb-impl)
Expand Down Expand Up @@ -557,6 +568,7 @@ The following software have components provided under the terms of this license:

- Guava: Google Core Libraries for Java (from http://code.google.com/p/guava-libraries, https://github.com/google/guava, https://repo1.maven.org/maven2/com/google/guava/guava)
- Netty/Codec (from http://netty.io/, https://repo1.maven.org/maven2/io/netty/netty-codec)
- Netty/Codec/Base (from https://netty.io/)

========================================================================
gpl-2.0-classpath
Expand Down
133 changes: 133 additions & 0 deletions devops/core-plus/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# OSDU Core Partition Deployment

## Overview

This directory deploys two components via Helm:

| Component | Purpose | Type | Lifecycle |
|-----------|---------|------|-----------|
| **Partition Service** | REST API for partition management | Java app | Always running |
| **Bootstrap** | Initializes partition properties | Bash scripts | Runs once on startup, then idles |

## Quick Start

```bash
cd deploy/
helm install core-partition-deploy .
```

See [Deploy README](./deploy/README.md) for full installation guide.

---

## Component Details

### Bootstrap

**What it does:**
- Calls Partition API to create/update partition config
- Sets storage, database, messaging, and feature flags
- Runs once on startup, then marks ready via `/tmp/bootstrap_ready` and remains idle

> **⚠️ DEPRECATION NOTICE:** MinIO configuration will be deprecated in the next release and replaced with SeaweedFS. Please plan your migration accordingly.

**Key Files:**
- Deployment: `deploy/templates/deploy-bootstrap.yaml`
- ConfigMap: `deploy/templates/configmap-bootstrap.yaml` (extensive config)
- Scripts: `bootstrap/bootstrap_partition.sh`

**Configuration (values.yaml):**
```yaml
data:
bootstrapImage: "" # Bootstrap image
bucketPrefix: "refi"
minioExternalEndpoint: ""

# Feature flags (used by bootstrap)
indexerAugmenterEnabled: "true"
asIngestedCoordinatesEnabled: "true"
keywordLowerEnabled: "true"
bagOfWordsEnabled: "true"
collaborationsEnabled: "true"
autocompleteEnabled: "false"
policyServiceEnabled: "false"
edsEnabled: "false"
```

**Full details:** [Bootstrap README](./bootstrap/README.md)

---

### Partition Service (Non-Bootstrap)

**What it does:**
- Provides partition CRUD API endpoints
- Runs continuously as a service
- Connects to PostgreSQL for metadata

**Key Files:**
- Deployment: `deploy/templates/deploy.yaml`
- ConfigMap: `deploy/templates/configmap.yaml` (LOG_LEVEL only)
- Service: `deploy/templates/service.yaml`
- Virtual Service: `deploy/templates/virtual-service.yaml`

**Configuration (values.yaml):**
```yaml
data:
image: "" # Service image
logLevel: "ERROR" # Application logging
requestsCpu: "5m"
requestsMemory: "350Mi"

conf:
appName: "partition"
replicas: 1
```

**Health Endpoints:**
- `/health/liveness` (port 8081)
- `/health/readiness` (port 8081)

---

## Configuration Summary

| Setting | Affects | Used In |
|---------|---------|---------|
| `data.image` | Partition Service | deploy.yaml |
| `data.bootstrapImage` | Bootstrap | deploy-bootstrap.yaml |
| `data.logLevel` | Partition Service | configmap.yaml |
| `data.bucket*`, `data.minio*` **(deprecated)** | Bootstrap | configmap-bootstrap.yaml |
| `data.*Enabled` (feature flags) | Bootstrap | configmap-bootstrap.yaml |
| `istio.proxyCPU*` | Partition Service | deploy.yaml |
| `istio.bootstrapProxyCPU*` | Bootstrap | deploy-bootstrap.yaml |

---

## Troubleshooting

**Bootstrap issues:**
```bash
# Check bootstrap logs
kubectl logs -l app=partition-bootstrap -f

# Verify completion
kubectl exec deployment/partition-bootstrap -- cat /tmp/bootstrap_ready
```

**Service issues:**
```bash
# Check service logs
kubectl logs -l app=partition -f

# Check health
kubectl exec deployment/partition -- curl localhost:8081/health/readiness
```

---

## Documentation

- **[Bootstrap Details](./bootstrap/README.md)** - Scripts and environment variables
- **[Helm Guide](./deploy/README.md)** - Installation and configuration
- **[Testing](./test/)** - Validation scripts
55 changes: 31 additions & 24 deletions devops/core-plus/bootstrap/README.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,27 @@
# Partition Bootstrapping Script

> **Part of:** [Core-Plus Deployment](../README.md) - runs as one-time initialization job

---

## Overview

This folder contains Bash scripts and a Dockerfile to bootstrap and configure a data partition. The scripts initialize partition properties by calling the Partition API, creating or updating the partition as needed.

## Key Environment Variables

| Environment Variable | Description |
| --------------------------------- | ---------------------------------------------------------- |
| `PARTITION_HOST` | Host for the partition API. |
| `DATA_PARTITION_ID` | Identifier for the data partition. |
| `BUCKET_PREFIX` | Prefix for bucket names used in the partition. |
| `PARTITION_SUFFIX` | Suffix used for partition-specific secret references. |
| `SERVICE_ACCOUNT` | Service account associated with the partition. |
| `SECRET_SERVICE_NAMESPACE` | Kubernetes namespace for the secret service. |
| `MINIO_ENDPOINT` | Endpoint for the MinIO service. |
| `MINIO_EXTERNAL_ENDPOINT` | External endpoint for accessing MinIO. |
| `MINIO_IGNORE_CERT_CHECK` | Flag to ignore SSL certificate checks for MinIO. |
| `MINIO_UI_ENDPOINT` | Endpoint for the MinIO user interface. |
| `ELASTIC_HTTPS` | Flag to enable HTTPS for Elasticsearch connections. |
| `INDEXER_AUGMENTER_ENABLED` | Flag to enable or disable the index augmenter. |
| `EDS_ENABLED` | Feature flag to enable or disable EDS. |
| `POLICY_SERVICE_ENABLED` | Feature flag to enable or disable OPA/policy enforcement. |
| `AUTOCOMPLETE_ENABLED` | Feature flag to enable or disable autocomplete. |
| `AS_INGESTED_COORDINATES_ENABLED` | Feature flag to enable or disable as-ingested coordinates. |
| `KEYWORD_LOWER_ENABLED` | Feature flag to enable or disable lowercase keywords. |
| `BAG_OF_WORDS_ENABLED` | Feature flag to enable or disable bag-of-words indexing. |
| `COLLABORATIONS_ENABLED` | Feature flag to enable or disable collaborations. |
## Environment Variables

### Variables for bootstrap_partition.sh

| Environment Variable | Description |
| -------------------- | ------------------------------- |
| `PARTITION_HOST` | Host for the partition API. |
| `DATA_PARTITION_ID` | Identifier for the data partition. |

### Variables for data_core.sh

The `data_core.sh` script uses additional environment variables to build the partition configuration JSON payload.

For the complete list of variables and their configuration, see the **Bootstrap Configuration** section in [Deploy README](../deploy/README.md).

## Scripts

Expand All @@ -48,4 +43,16 @@ This script defines the `core_partition_data` function, which outputs the full J

## Docker

The `Dockerfile` builds an Alpine-based image that runs `bootstrap_partition.sh` on startup and then sleeps to keep the container alive. Required environment variables must be supplied at runtime (e.g. via Kubernetes environment or Docker `--env`/`--env-file`).
The `Dockerfile` builds an Alpine-based image that runs `bootstrap_partition.sh` on startup and then sleeps to keep the container alive. Required environment variables must be supplied at runtime (e.g. via Kubernetes environment or Docker `--env`/`--env-file`).

---

## Deployment Integration

This bootstrap runs automatically via Helm deployment:
- **Image:** Specified in `values.yaml` → `data.bootstrapImage`
- **ConfigMap:** `partition-config-bootstrap` (see [template](../deploy/templates/configmap-bootstrap.yaml))
- **Service Account:** `partition-bootstrap`
- **Readiness:** Checks `/tmp/bootstrap_ready` file

See [Deploy README](../deploy/README.md) for installation.
Loading