1.6.2 sync fixed

[CathalOConnorRH]

Why are these changes needed?

Related issue number

Checks

• I've made sure the tests are passing.
• Testing Strategy
  • Unit tests
  • Manual tests
  • This PR is not tested :(

[pawelpaszki]

there might be some missing updates, which we can fix once we try to merge the target branch onto dev, but the immediate one that I can see is missing entry in the .gitignore: scripts/update-resources.

I'll try to verify the PR shortly

[pawelpaszki]

✅ OK to merge to rhoai-3.5-GA-release — address outstanding items before merging to dev

Review focused on the upstream v1.6.2 sync and ODH e2e/containerised test setup. This PR is reasonable to merge onto rhoai-3.5-GA-release for initial verification, but the follow-up rhoai-3.5-GA-release → dev integration PR should deliberately resolve the items below (including reconciliation with an outdated dev base and companion odh-konflux-central updates).

---

Executive summary

Large upstream sync (375 commits, ~115k additions) from v1.4.2 → v1.6.2. ODH-specific work on top includes e2e path updates, matrix PR e2e runner, auth controller OpenShift guard, and v1.6.2 cherry-pick.

GitHub Actions PR e2e subset is green (all 4 matrix jobs pass). Main gaps are in the containerised / Konflux e2e path and in reconciling with current dev.

Check	Status (last run)
Lint, build, helm/codegen verify	✅ Pass
e2e-rayjob (4 matrix jobs)	✅ Pass
Operator Chaos	❌ Fail — 1 breaking CRD change
CodeRabbit	Skipped (713 files > 150 limit)

---

E2E test setup

1. GitHub Actions PR e2e (Kind) — ✅ good

The refactor (scripts/run-e2e-rayjob-kind.sh, build-once + matrix) looks solid:

• Build once, test in parallel via artifact + matrix
• One test per invocation avoids OOM/concurrency issues with TestRayJobWithClusterSelector
• Correct package path: ./test/e2erayjob
• Go 1.25, Ray image rayproject/ray:2.52.1

2. Container test image (quay.io/opendatahub/kuberay-tests) — partially updated

Updated correctly in this PR:

• run-tests.sh tier → package mapping for upstream split layout
• Smoke tier renamed auth test: TestRayClusterAuthentication → TestRayClusterAuthOptions
• Tier1 timeout bumped to 60m (autoscaler included)
• Root Makefile paths updated for test/support/support.go and test/e2erayjob/

Outstanding issues (fix before / during dev merge):

Issue	Severity	Detail
Go version mismatch in test Dockerfile	High	ray-operator/images/tests/Dockerfile uses golang:1.24 but go.mod requires 1.25 → go mod download fails in container build (GOTOOLCHAIN=local). Bump to golang:1.25.
Missing autoscaler resource patch in Makefile	High	Current dev patches ray-operator/test/e2eautoscaler/support.go for build-image; this PR drops that step while Tier1 runs ./test/e2eautoscaler. Restore with new paths during dev merge.
.env-odh Ray image	Medium	Still pins an older quay.io/modh/ray digest. Smoke tier runs token auth tests requiring Ray ≥ 2.52. GH Actions uses rayproject/ray:2.52.1; container tests may not. Reconcile with dev's digest or update to verified 2.52+.
odh-release.yml stale path	Low	Still compiles ./test/e2e/ — may produce incomplete/wrong release artifacts.

3. odh-konflux-central — will break once this lands on dev

Pipeline still references pre-upstream-sync paths:

go run ../scripts/update-resources.go -scenario build-image test/e2e/support.go

After this sync lands on dev, update to at minimum:

go run ../scripts/update-resources.go -scenario build-image test/support/support.go
go run ../scripts/update-resources.go -scenario build-image test/e2erayjob/rayjob_lightweight_test.go
go run ../scripts/update-resources.go -scenario build-image test/e2eautoscaler/support.go

Note: Konflux ITS is configured for on-target-branch: [dev], so it won't run against this PR's target (rhoai-3.5-GA-release) — final containerised verification belongs on the dev integration PR, alongside rebuilding/pushing kuberay-tests:latest.

Existing odh-konflux-central branch work (kuberay-e2e-more-fixes) has SCC//tmp clone/must-gather improvements but not the path migration above.

---

Other notable findings

Operator Chaos failure — expected, needs decision

Fails on one breaking CRD change:

rayjobs.ray.io v1 — FieldRemoved (Breaking): .spec.deletionPolicy
replaced by .spec.deletionStrategy

Intentional upstream API evolution for v1.6.x. For a 1.4.2 → 1.6.2 jump, either update chaos/knowledge/kuberay.yaml or document a bootstrap waiver.

ODH midstream fix — looks good

AuthenticationController guards Route ownership behind IsOpenShift — sensible for Kind/non-OCP tests.

PR hygiene

• Description empty; testing checklist unchecked despite green e2e
• Branch name 1.6.1-sync-fixed vs title/content 1.6.2
• CodeRabbit skipped due to size — manual review of CRD/RBAC/auth/webhook midstream code still warranted before GA

---

Recommended action plan

Merge now (this PR → rhoai-3.5-GA-release)

• Upstream sync + GH Actions Kind e2e verification
• Do not block on Konflux ITS (targets dev only)

Before / during rhoai-3.5-GA-release → dev integration PR

In kuberay:

1. Bump ray-operator/images/tests/Dockerfile to golang:1.25
2. Restore autoscaler patching in root Makefile with new paths
3. Verify/update .env-odh to Ray 2.52+ (reconcile with current dev digest)
4. Fix or deprecate odh-release.yml compile path
5. Update chaos/knowledge/kuberay.yaml for v1.6.2 CRDs (or document operator-chaos waiver)

In odh-konflux-central (companion PR):

1. Update integration-tests/kuberay/pr-testing-pipeline.yaml resource patch paths
2. Merge/rebase existing konflux kuberay fixes
3. Rebuild and push kuberay-tests:latest
4. Run full Tier1+Smoke ITS on a PR targeting dev

---

Verdict

Approve for merge to rhoai-3.5-GA-release. Treat the path to dev as the definitive integration gate — especially containerised/Konflux e2e, test-image rebuild, and reconciliation with the outdated dev base.

[pawelpaszki]

Sync review: ODH carry dropped during 1.6.x merge

While validating Tier1/Smoke against this branch (custom ImageStream deploy, Ray set to Removed), several auth-related e2e failures looked genuine, not flukes from unmanaged Ray.

Failures observed

• TestRayClusterAuthOptions / RayJob lightweight auth: RayCluster becomes Ready, but ray-head has no RAY_AUTH_MODE
• TestRayJob/Successful_RayJob: counter-head-svc not found (weaker signal; may be secondary/flake)
• Most non-auth tests passed (autoscaler, lightweight RayJob success/fail/stop, suspend, cluster selector)

Important distinction

There are two different auth models:

1. ODH / OpenShift auth (midstream carry): odh.ray.io/secure-trusted-network + kube-rbac-proxy sidecar via AuthenticationController
2. Upstream Ray token auth (1.6.x): spec.authOptions.mode=token → RAY_AUTH_MODE / RAY_AUTH_TOKEN

Base (rhoai-3.5-GA-release) e2e tested (1). This PR replaced that with upstream tests for (2). kube-rbac-proxy in failing-test logs is (1), not proof that (2) was applied.

---

What was dropped (not whole files — call sites / wiring)

These helpers still exist on both base and this branch:

• ray-operator/controllers/ray/utils/auth_sidecar.go (InjectAuthSidecar)
• ray-operator/controllers/ray/authentication_controller.go (GetOIDCProxySidecar / GetOIDCProxyVolumes)
• mutating webhook / mTLS / networkpolicy carries

What the sync removed from ray-operator/controllers/ray/raycluster_controller.go (vs rhoai-3.5-GA-release):

1. AuthenticationReady gate before head pod create
   Base ~lines 1158–1188 — wait for auth SA / AuthenticationReady before calling buildHeadPod.

2. InjectAuthSidecar in buildHeadPod
   Base ~lines 1261–1282 — inject kube-rbac-proxy via:

   utils.InjectAuthSidecar(
       &podConf.Spec,
       &instance,
       authMode,
       GetOIDCProxySidecar,
       GetOIDCProxyVolumes,
       namer.ServiceAccountName(authMode),
   )

   This call site is absent from current buildHeadPod on this branch.

3. ODH auth e2e overwritten in ray-operator/test/e2e/raycluster_auth_test.go

   • Base: TestRayClusterAuthentication (sidecar / OpenShift auth)
   • This PR: TestRayClusterAuthOptions (upstream Ray token auth only)

---

Suggestions

1. Re-carry the AuthenticationReady gate and InjectAuthSidecar wiring from base into current raycluster_controller.go, alongside upstream reconcileAuthSecret + configureTokenAuth / AuthOptions.
2. Keep both e2e covers:
   • ODH OpenShift auth (TestRayClusterAuthentication / kube-rbac-proxy)
   • Upstream token auth (authOptions.mode=token → RAY_AUTH_*)
3. Rebuild + redeploy the operator from the fixed tree, then re-run the failing auth tests (and ideally the old ODH auth test).

Priority file: ray-operator/controllers/ray/raycluster_controller.go — helpers are still present; the sync stopped calling them.

[pawelpaszki]

Verified by building and deploying kuberay to a cluster and running all e2e tests that we usually run (tier1 + smoke). all good