Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
284 changes: 284 additions & 0 deletions docs/api/v2-clientgo-vs-custom-sdk.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,284 @@
# Client-Go for rosa-hyperfleet-api: Why It Works and How to Do It

> **Generation prompt:** `/tmp/prompt-lib/v2.p1.md` — explains how to build the
> rosa-hyperfleet-api with client-go support instead of a custom SDK like
> ocm-sdk-go, using the hypershift project as a reference.

---

## The Problem

External Go programs need to interact with rosa-hyperfleet-api resources
(Cluster, NodePool, ManagementCluster, Manifest, Placement). The traditional
OCM approach is to build a bespoke SDK (`ocm-sdk-go`) with hand-written
request/response builders, connection management, authentication helpers, and
serialization logic — hundreds of generated files that must be versioned and
maintained in lockstep with the API.

**The question:** Can we skip all that and let consumers use standard
Kubernetes client-go tooling instead?

**The answer:** Yes — and the project is already 80% of the way there.

---

## How HyperShift Does It

The [openshift/hypershift](https://github.com/openshift/hypershift) project
publishes a **standalone API types module** (`github.com/openshift/hypershift/api`)
and a **generated typed clientset** (`github.com/openshift/hypershift/client`).

External consumers have two choices:

### Option A: controller-runtime client (lightweight)

```go
import (
hyperv1 "github.com/openshift/hypershift/api/hypershift/v1beta1"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/scheme"
)

// Register the scheme once
s := runtime.NewScheme()
hyperv1.AddToScheme(s)

// Use any controller-runtime client
c, _ := client.New(cfg, client.Options{Scheme: s})
c.List(ctx, &hyperv1.HostedClusterList{}, client.InNamespace("clusters"))
```
Comment thread
cdoan1 marked this conversation as resolved.

### Option B: native typed clientset (familiar kubectl-style API)

```go
import (
hyperclient "github.com/openshift/hypershift/client/clientset/clientset"
)

cs, _ := hyperclient.NewForConfig(cfg)
hcs, _ := cs.HypershiftV1beta1().HostedClusters("clusters").List(ctx, metav1.ListOptions{})
```

Both work because hypershift runs `k8s.io/code-generator` (`client-gen`,
`lister-gen`, `informer-gen`, `applyconfiguration-gen`) via a
`hack/update-codegen.sh` script.

---

## What rosa-hyperfleet-api Already Has

| Piece | Status | Where |
|-------|--------|-------|
| Standalone API types module | **Done** | `hyperfleet-operator/api/go.mod` |
| Minimal dependencies on API module | **Done** | Only `k8s.io/apimachinery` + `openshift/hypershift/api` |
| `SchemeBuilder` + `AddToScheme()` | **Done** | `api/v1alpha1/groupversion_info.go` |
| `+kubebuilder:object:root=true` markers | **Done** | All 5 CRD types |
| Deep-copy generation | **Done** | `zz_generated.deepcopy.go` |
| CRD YAML manifests | **Done** | `config/crd/bases/` |
| Generated typed clientset | **Not done** | — |
| Generated informers / listers | **Not done** | — |

**The API types module is already structured correctly.** Any Go program can
import `github.com/openshift-online/rosa-hyperfleet-api/hyperfleet-operator/api`
today with zero extra work and use it with controller-runtime's `client.Client`.

---

## What It Takes to Add Full client-go Support

### Step 1: controller-runtime path (works today, zero changes)

An external Go consumer can already do this:

```go
import (
hyperfleetv1 "github.com/openshift-online/rosa-hyperfleet-api/hyperfleet-operator/api/v1alpha1"
"k8s.io/apimachinery/pkg/runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/client/config"
)

func main() {
s := runtime.NewScheme()
hyperfleetv1.AddToScheme(s)

cfg := config.GetConfigOrDie()
c, err := client.New(cfg, client.Options{Scheme: s})

// List clusters
clusters := &hyperfleetv1.ClusterList{}
c.List(ctx, clusters, client.InNamespace("my-namespace"))

// Create a node pool
np := &hyperfleetv1.NodePool{
ObjectMeta: metav1.ObjectMeta{Name: "pool-1", Namespace: "cluster-uuid"},
Spec: hyperfleetv1.NodePoolSpec{...},
}
c.Create(ctx, np)

// Watch for changes
c.Watch(ctx, &hyperfleetv1.ClusterList{})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== Relevant doc excerpt =="
sed -n '105,130p' docs/api/v2-clientgo-vs-custom-sdk.md | cat -n

echo
echo "== Controller-runtime version =="
if [ -f go.mod ]; then
  rg -n 'sigs\.k8s\.io/controller-runtime' go.mod || true
fi

echo
echo "== Any mention of Watch-capable client in repo =="
rg -n 'NewWithWatch|WithWatch|Watch\(' docs go.mod .github -g '!**/vendor/**' || true

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 1430


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== File size =="
wc -l docs/api/v2-clientgo-vs-custom-sdk.md

echo
echo "== Lines around the cited snippet =="
sed -n '115,125p' docs/api/v2-clientgo-vs-custom-sdk.md | cat -n

echo
echo "== Search for any local helper/interface adding Watch to client.Client =="
rg -n 'type .*interface.*Watch|interface .*Watch|func .*Watch\(' -g '*.go' . || true

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 809


Use a watch-capable client here. client.New returns client.Client, which doesn’t expose Watch; switch this example to client.NewWithWatch or wrap it with client.WithWatch so the snippet is valid.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/api/v2-clientgo-vs-custom-sdk.md` at line 120, Update the client
initialization in the example to use a watch-capable client via
client.NewWithWatch or client.WithWatch before calling c.Watch. Keep the
existing Watch invocation and ClusterList argument unchanged.

}
```

This is the **recommended starting point** — it requires no code generation,
no new packages, and no maintenance burden.

### Step 2: native typed clientset (optional, for kubectl-like ergonomics)

If consumers want typed methods like
`Clusters("ns").Get(ctx, "name", metav1.GetOptions{})` instead of the
generic `client.Get(ctx, key, &obj)`, add code-generator:

```bash
# 1. Add code-generator to hack/tools
go get k8s.io/code-generator@v0.36.0

# 2. Add generation tags to api/v1alpha1/doc.go
// +k8s:deepcopy-gen=package,register
// +k8s:defaulter-gen=TypeMeta
// +k8s:openapi-gen=true
// +groupName=hyperfleet.io

# 3. Create hack/update-codegen.sh (mirror hypershift's script)
# Generates: client/clientset/, client/informers/, client/listers/

# 4. Add Makefile target
make clients
```
Comment on lines +133 to +148

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the referenced doc and inspect the relevant section with line numbers.
doc="docs/api/v2-clientgo-vs-custom-sdk.md"
echo "=== $doc (lines 120-170) ==="
sed -n '120,170p' "$doc" | cat -n

echo
echo "=== Search for Kubernetes dependency/version pins ==="
rg -n --hidden --glob '!**/.git/**' 'k8s.io/(client-go|apimachinery|code-generator)|code-generator@|client-go' -S . | sed -n '1,220p'

echo
echo "=== Search for genclient markers and related codegen notes ==="
rg -n --hidden --glob '!**/.git/**' '\+genclient|\+k8s:deepcopy-gen|\+k8s:defaulter-gen|\+k8s:openapi-gen|hack/update-codegen|make clients' -S . | sed -n '1,220p'

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 24495


🏁 Script executed:

#!/bin/bash
set -euo pipefail

doc="docs/api/v2-clientgo-vs-custom-sdk.md"
echo "=== $doc (lines 120-170) ==="
sed -n '120,170p' "$doc" | cat -n

echo
echo "=== Search for Kubernetes dependency/version pins ==="
rg -n --hidden --glob '!**/.git/**' 'k8s.io/(client-go|apimachinery|code-generator)|code-generator@|client-go' -S . | sed -n '1,220p'

echo
echo "=== Search for genclient markers and related codegen notes ==="
rg -n --hidden --glob '!**/.git/**' '\+genclient|\+k8s:deepcopy-gen|\+k8s:defaulter-gen|\+k8s:openapi-gen|hack/update-codegen|make clients' -S . | sed -n '1,220p'

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 24495


🏁 Script executed:

#!/bin/bash
set -euo pipefail

doc="docs/api/v2-clientgo-vs-custom-sdk.md"
echo "=== $doc (lines 120-170) ==="
sed -n '120,170p' "$doc" | cat -n

echo
echo "=== Candidate dependency manifest files ==="
git ls-files | rg '(^|/)(go\.mod|go\.sum|Makefile|go\.work|package\.json|buf\.yaml|helmfile\.yaml|\.mod)$|(^|/)(api|apis|pkg)/.*\.go$' | sed -n '1,240p'

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 4981


🌐 Web query:

Kubernetes code-generator version must match client-go/apimachinery version genclient markers required typed clients

💡 Result:

When working with Kubernetes code generation, maintaining version synchronization between components is critical for stability, though the specific mechanisms and requirements are well-defined: 1. Version Matching: The k8s.io/code-generator repository explicitly states that its HEAD matches the HEAD of k8s.io/apiserver, k8s.io/apimachinery, and k8s.io/client-go [1][2][3]. When consuming these libraries, it is standard practice to align the versions of k8s.io/code-generator, k8s.io/apimachinery, and k8s.io/client-go within your go.mod file [2][4]. Using mismatched versions often leads to compilation errors or subtle runtime issues due to changes in internal API structures or helper library signatures [5]. 2. Genclient Markers: To generate typed clients for your API types, you must use the // +genclient marker [6][7]. - This marker must be placed above the struct definition for the API type (e.g., in your types.go file) [6]. - It must not be placed above the List type (e.g., PodList) [7]. - The generator uses this marker as an opt-in mechanism; without it, the client-gen tool will ignore the type [7][8]. - Additional tags such as // +genclient:nonNamespaced, // +genclient:noStatus, and various verb-control tags (e.g., // +genclient:onlyVerbs=...) can be used to further configure the behavior of the generated client [6][7]. In summary, ensure that your k8s.io dependencies are pinned to compatible versions—ideally matching the Kubernetes release cycle you are targeting [5]—and ensure that every resource you wish to have a typed client for is explicitly annotated with the // +genclient tag [6][7].

Citations:


🌐 Web query:

site:kubernetes.io code-generator genclient marker typed client client-gen README version match client-go/apimachinery

💡 Result:

In the context of the Kubernetes code-generator toolset, the +genclient marker is a directive used to instruct client-gen to generate a typed client for a specific API group or resource [1]. Key points regarding the use of these markers and version matching include: 1. Marker Purpose: The +genclient marker is typically placed in the doc.go file or above the relevant type definition within your API package to signal that the code generator should produce a typed clientset for the associated custom resource [1]. This enables type-safe interactions with your Kubernetes custom resources rather than relying on less efficient or error-prone dynamic clients [1]. 2. Version Matching (client-go and code-generator): It is critical that the version of k8s.io/code-generator matches the version of k8s.io/client-go and the Kubernetes API machinery your project depends on [1]. Mismatches between these dependencies often lead to build failures, incompatible generated code, or runtime errors because the internal structures (such as API schemes and object metadata) may differ between versions [1][2]. 3. Implementation Best Practices: - When setting up your project, ensure that the version (or branch) of code-generator matches the target Kubernetes release [1]. - If using Go modules, the go.mod file should reflect consistent versions across all k8s.io/* repositories. Historically, projects pinned these to specific Kubernetes release tags (e.g., kubernetes-1.9.0) to ensure compatibility [1]. - Beyond genclient, modern Kubernetes development also utilizes various other markers (e.g., +k8s:listType, +k8s:immutable) for declarative validation and server-side apply, which are processed by specialized generators like validation-gen [3][4][5]. Always consult the go.mod file of the specific kubernetes/kubernetes release you are targeting to ensure that your local development dependencies align with the upstream source [6][2].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== api/v1alpha1 files ==="
git ls-files 'hyperfleet-operator/api/v1alpha1/*' | sed -n '1,200p'

echo
echo "=== api/v1alpha1/doc.go ==="
sed -n '1,220p' hyperfleet-operator/api/v1alpha1/doc.go | cat -n

echo
echo "=== api/v1alpha1/cluster_types.go ==="
sed -n '1,220p' hyperfleet-operator/api/v1alpha1/cluster_types.go | cat -n

echo
echo "=== api/v1alpha1/nodepool_types.go ==="
sed -n '1,220p' hyperfleet-operator/api/v1alpha1/nodepool_types.go | cat -n

echo
echo "=== api/v1alpha1/placement_types.go ==="
sed -n '1,220p' hyperfleet-operator/api/v1alpha1/placement_types.go | cat -n

echo
echo "=== api/v1alpha1/managementcluster_types.go ==="
sed -n '1,220p' hyperfleet-operator/api/v1alpha1/managementcluster_types.go | cat -n

echo
echo "=== api/v1alpha1/manifest_types.go ==="
sed -n '1,220p' hyperfleet-operator/api/v1alpha1/manifest_types.go | cat -n

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 689


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== types files with line numbers ==="
for f in \
  hyperfleet-operator/api/v1alpha1/cluster_types.go \
  hyperfleet-operator/api/v1alpha1/nodepool_types.go \
  hyperfleet-operator/api/v1alpha1/placement_types.go \
  hyperfleet-operator/api/v1alpha1/managementcluster_types.go \
  hyperfleet-operator/api/v1alpha1/manifest_types.go \
  hyperfleet-operator/api/v1alpha1/groupversion_info.go
do
  echo
  echo "--- $f ---"
  sed -n '1,220p' "$f" | cat -n
done

echo
echo "=== search for genclient markers in repo ==="
rg -n --hidden --glob '!**/.git/**' '\+genclient|client-gen|customresourcedefinitions|SchemeBuilder|AddToScheme' hyperfleet-operator/api platform-api hyperfleet-db docs -S | sed -n '1,240p'

Repository: openshift-online/rosa-hyperfleet-api

Length of output: 28947


Add +genclient on the concrete API types.
Package-level tags alone won’t generate a typed clientset; Cluster, NodePool, Placement, ManagementCluster, and Manifest need +genclient markers.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/api/v2-clientgo-vs-custom-sdk.md` around lines 133 - 148, Update the API
type declarations for Cluster, NodePool, Placement, ManagementCluster, and
Manifest to include +genclient markers, while preserving the existing
package-level generation tags and client generation workflow.


After generation, external usage becomes:

```go
import (
hfclient "github.com/openshift-online/rosa-hyperfleet-api/client/clientset/clientset"
)

cs, _ := hfclient.NewForConfig(cfg)

// Typed, discoverable API
cluster, _ := cs.HyperfleetV1alpha1().Clusters("ns").Get(ctx, "my-cluster", metav1.GetOptions{})
pools, _ := cs.HyperfleetV1alpha1().NodePools("ns").List(ctx, metav1.ListOptions{})
```

---

## Comparison: client-go vs ocm-sdk-go Style

| Dimension | client-go (hypershift model) | Custom SDK (ocm-sdk-go model) |
|-----------|------------------------------|-------------------------------|
| **Client code complexity** | 5–10 lines to get a working client | Connection builders, token refresh, pagination helpers, custom types |
| **Server-side complexity** | Kubernetes API server handles serialization, validation, watch, RBAC | Must build all REST handlers, pagination, filtering, auth middleware |
| **Type sharing** | Import the API module directly — one source of truth | Duplicate types in SDK, keep in sync manually |
| **Authentication** | kubeconfig, service accounts, OIDC — all built-in | Custom token flow, SigV4, or bespoke auth |
| **Authorization** | Kubernetes RBAC, built-in | Custom (Cedar/AVP in this project) |
| **Watch / real-time** | Built-in (`Watch`, informers, work queues) | Must build SSE/websocket/polling |
| **Generated client maintenance** | `make clients` reruns code-gen | Maintain SDK generator + SDK repo |
| **Ecosystem compatibility** | Works with kubectl, kustomize, ArgoCD, Flux, any K8s tooling | Only works with the custom SDK |
| **Learning curve for consumers** | Standard Kubernetes patterns — most Go developers already know this | Must learn the SDK's API surface |
| **Go dependency weight** | `k8s.io/client-go` (~well-known, already in most K8s projects) | Custom SDK dependency |

---

## The Critical Architectural Question

There is one important distinction to understand:

**HyperShift's CRDs run on a real Kubernetes API server.** Consumers point
their kubeconfig at a management cluster and talk to it using standard
Kubernetes API machinery (etcd-backed, with RBAC, watches, etc.).

**rosa-hyperfleet-api's platform-api is a REST gateway** that does NOT expose
a Kubernetes API server endpoint. It uses `hyperfleet-db` (PostgreSQL) as its
backing store and speaks REST/JSON over an API Gateway with SigV4 auth and
Cedar/AVP authorization.

This means:

| Scenario | client-go works? | Notes |
|----------|-------------------|-------|
| **Operator-to-operator** (inside the K8s cluster) | **Yes, today** | The hyperfleet-operator already uses controller-runtime against the API server where CRDs are installed |
| **External tool → K8s API server** (if CRDs are exposed) | **Yes, today** | Any kubeconfig-holding client can CRUD the CRDs directly |
| **External tool → platform-api REST gateway** | **No** | platform-api speaks REST, not the Kubernetes API protocol — client-go cannot target it directly |

### What This Means In Practice

If the goal is "external Go clients can manage hyperfleet resources," there are
two clean paths:

#### Path 1: Expose CRDs on a Kubernetes API Server (simplest)

Let external clients talk directly to the management cluster's Kubernetes API
where the CRDs are installed. This is what hypershift does.

- **Pros:** Zero SDK work. client-go just works. kubectl just works.
Full watch/informer support. RBAC for free.
- **Cons:** Requires the client to have a kubeconfig for the management
cluster. Authorization is Kubernetes RBAC, not Cedar/AVP.

#### Path 2: Keep the REST Gateway, Use the Types Module Only

Keep platform-api as the entry point, but let Go clients import the API types
module for struct definitions and serialization:

```go
import (
hyperfleetv1 "github.com/openshift-online/rosa-hyperfleet-api/hyperfleet-operator/api/v1alpha1"
"encoding/json"
"net/http"
)

cluster := &hyperfleetv1.Cluster{
Spec: hyperfleetv1.ClusterSpec{...},
}
body, _ := json.Marshal(cluster)
req, _ := http.NewRequest("POST", "https://api.example.com/v1/clusters", bytes.NewReader(body))
// Add SigV4 signing ...
```

This is simpler than a full SDK because the types are already published —
consumers just marshal/unmarshal them. But they don't get the rich client-go
experience (typed methods, watches, informers).

#### Path 3: API Server Aggregation or Virtual API Server (advanced)

Implement a Kubernetes API server facade in front of hyperfleet-db so that the
platform-api speaks the Kubernetes API protocol. Then client-go works end to
end. Projects like `kcp` and `crossplane` do this.

- **Pros:** Full client-go ecosystem support against the managed API.
- **Cons:** Significant engineering effort to implement the API server
protocol (discovery, OpenAPI, watch, pagination, field selectors).

---

## Recommendation

**Start with Path 1 + the controller-runtime pattern.** The project already
has everything needed:

1. The `hyperfleet-operator/api` module is standalone and lightweight
2. `AddToScheme()` is implemented
3. CRDs are generated and installable
4. Any Go client can `import` the types and use `controller-runtime/pkg/client`

For clients that need to go through the REST gateway (SigV4/Cedar auth), use
**Path 2** — publish the types module and let consumers use it for
serialization with a thin HTTP client. This is dramatically simpler than
building an ocm-sdk-go-style SDK.
Comment thread
cdoan1 marked this conversation as resolved.

Only invest in **generated typed clientsets** (code-generator) if there is
strong demand from consumers who prefer the `Clientset.Resource().Verb()`
pattern over controller-runtime's generic client.

---

## Summary

| Question | Answer |
|----------|--------|
| Can we use client-go instead of a custom SDK? | **Yes** — for direct K8s API access, it works today with zero changes |
| Is it simpler? | **Significantly** — no SDK to build, version, or maintain |
| Is the client code simpler? | **Yes** — 5–10 lines vs. connection builders + pagination + auth helpers |
| Does it work for the REST gateway? | **Partially** — types can be shared, but client-go cannot target a non-K8s REST API directly |
| What should we do first? | Publish the API types module, document the controller-runtime usage pattern, and optionally run code-generator for typed clientsets |
Comment thread
cdoan1 marked this conversation as resolved.