fix(ec2): add error logging for credential failures in EC2 discovery

[wkrause13]

When an EC2 upstream is configured with invalid or missing AWS credentials, the discovery would fail silently without clear error logs. This made it difficult to diagnose issues where EC2 instances became inaccessible.

This change adds explicit ERROR level logging in eds.go when:

• Secret listing fails for EC2 upstream discovery
• EC2 instance fetching fails (e.g., invalid credentials, access denied)

The error message now clearly indicates "failed to fetch EC2 instances for upstream discovery" with the underlying error details.

Description

API changes

Code changes

CI changes

Docs changes

Context

Interesting decisions

Testing steps

Testing EC2 Credential Error Logging

This document describes how to test the error logging fix for EC2 upstream discovery when AWS credentials are invalid or missing.

Background

When an EC2 upstream is configured with invalid or empty secretRef, the discovery fails silently without clear error logs. This fix adds explicit ERROR-level logging in eds.go to make credential failures visible.

Prerequisites

• Docker Desktop (or Docker daemon running)
• kind (Kubernetes in Docker)
• kubectl
• glooctl (or helm)
• Go 1.21+ (for building)

# Verify prerequisites
docker version
kind version
kubectl version --client
glooctl version
go version

Step 1: Create a Kind Cluster

# Start Docker if not running
open -a Docker  # macOS
# or: systemctl start docker  # Linux

# Wait for Docker to be ready
docker info >/dev/null 2>&1 || echo "Docker not ready"

# Create kind cluster
kind create cluster --name gloo-test --wait 120s

Step 2: Install Gloo OSS (Released Version)

This step installs the current released version to observe the baseline behavior.

glooctl install gateway

# Wait for pods to be ready
kubectl -n gloo-system get pods -w

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
discovery-xxx                    1/1     Running   0          1m
gateway-proxy-xxx                1/1     Running   0          1m
gloo-xxx                         1/1     Running   0          1m

Step 3: Create EC2 Upstream with Invalid Credentials

Create an EC2 upstream with empty secretRef to trigger the error condition:

kubectl apply -f - <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: test-ec2-upstream
  namespace: gloo-system
spec:
  awsEc2:
    region: us-east-1
    secretRef:
      name: ""
      namespace: ""
    filters:
      - key: "Name"
EOF

Step 4: Observe Baseline Behavior (Before Fix)

Wait a few seconds for the EDS polling to trigger, then check logs:

# Check gloo logs for EC2-related messages
kubectl -n gloo-system logs deployment/gloo | grep -iE "(ec2|aws|credential|error)" | tail -20

Expected baseline behavior:

• Only a WARN level log with unclear message: "received error and cannot aggregate it"
• Source: discovery/discovery.go:215
• No ERROR level log from the EC2 plugin itself

# Verify glooctl check shows no problems (this is the issue!)
glooctl check

Expected: "No problems detected" even though the upstream is broken.

Step 5: Build Patched Gloo Image

Build Gloo with the error logging fix:

cd /path/to/gloo  # Your gloo repository

# Verify the code change is in place
git diff projects/gloo/pkg/plugins/aws/ec2/eds.go

# Build and load into kind
CLUSTER_NAME=gloo-test make kind-build-and-load-gloo

This will:

1. Build the gloo binary with your changes
2. Build a Docker image with tag quay.io/solo-io/gloo:1.0.1-dev
3. Load the image into the kind cluster

Step 6: Deploy Patched Image

Update the gloo deployment to use the patched image:

# Patch the deployment
kubectl -n gloo-system set image deployment/gloo gloo=quay.io/solo-io/gloo:1.0.1-dev

# Wait for rollout
kubectl -n gloo-system rollout status deployment/gloo --timeout=120s

# Verify the new pod is running
kubectl -n gloo-system get pods -l gloo=gloo

Step 7: Verify the Fix

Check logs for the new ERROR-level logging:

# Check for ERROR logs from eds.go
kubectl -n gloo-system logs deployment/gloo | grep '"level":"error"' | head -5

Expected output with fix:

{
  "level": "error",
  "caller": "ec2/eds.go:104",
  "msg": "failed to fetch EC2 instances for upstream discovery",
  "error": "unable to get aws client: unable to create a session with credentials taken from secret ref: secrets not found for secret ref .: list did not find secret ."
}

Key improvements:

Aspect	Before	After
Log Level	warn	error
Message	"received error and cannot aggregate it"	"failed to fetch EC2 instances for upstream discovery"
Source	discovery/discovery.go	ec2/eds.go:104

The error will repeat every 30 seconds (EDS polling interval), providing continuous visibility.

Step 8: Test with Real AWS Credentials (Optional)

To test with real but invalid AWS credentials:

# Create a secret with invalid credentials
kubectl -n gloo-system create secret generic invalid-aws-creds \
  --from-literal=access_key=INVALID_ACCESS_KEY \
  --from-literal=secret_key=INVALID_SECRET_KEY

# Update the upstream to use the secret
kubectl apply -f - <<EOF
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: test-ec2-upstream
  namespace: gloo-system
spec:
  awsEc2:
    region: us-east-1
    secretRef:
      name: invalid-aws-creds
      namespace: gloo-system
    filters:
      - key: "Name"
EOF

# Check logs for the error
kubectl -n gloo-system logs deployment/gloo | grep '"level":"error"' | tail -5

This should show an error about AWS rejecting the credentials.

Cleanup

# Delete the kind cluster
kind delete cluster --name gloo-test

Notes for reviewers

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[solo-changelog-bot]

Issues linked to changelog:
https://github.com/solo-io/gloo/issues/0000