Lambda: spec.layers never reconciled when the function has no layers in AWS

[suprahh]

Describe the bug

When a Lambda Function has no layers attached in AWS, adding layers to spec.layers
has no effect. The controller never computes a Spec.Layers delta, so
UpdateFunctionConfiguration is called without the Layers parameter and the layers are
never attached. No error is reported and the resource is marked ACK.ResourceSynced=True.

The root cause is in the generated sdkFind for Function. It starts from a deep copy of
the desired resource and only overwrites Spec.Layers when the AWS response contains
layers:

// pkg/resource/function/sdk.go
ko := r.ko.DeepCopy()                        // starts from DESIRED state
...
if resp.Configuration.Layers != nil {
    ...
    ko.Spec.Layers = layer                   // only assigned when AWS already has layers
    ko.Status.LayerStatuses = f16
} else {
    ko.Status.LayerStatuses = nil            // clears STATUS only; Spec.Layers is left as-is
}

When AWS returns no layers, the else branch clears only Status.LayerStatuses. The
latest resource therefore keeps the desired Spec.Layers value inherited from the
DeepCopy. newResourceDelta then compares desired against latest:

// pkg/resource/function/delta.go
if len(a.ko.Spec.Layers) != len(b.ko.Spec.Layers) {
    delta.Add("Spec.Layers", a.ko.Spec.Layers, b.ko.Spec.Layers)
} else if len(a.ko.Spec.Layers) > 0 {
    if !ackcompare.SliceStringPEqual(a.ko.Spec.Layers, b.ko.Spec.Layers) {
        delta.Add("Spec.Layers", a.ko.Spec.Layers, b.ko.Spec.Layers)
    }
}

Both sides hold the same value, so no delta is added. updateFunctionConfiguration guards
the field with delta.DifferentAt("Spec.Layers"), which is never true, so input.Layers
is never set.

The bug is self-sealing: it only manifests in the exact case where the layers are missing
from AWS and therefore need to be added. If the function already has a different layer,
resp.Configuration.Layers != nil holds, Spec.Layers is correctly overwritten from the
response, the delta fires and the update works as expected.

Status.LayerStatuses is correctly reported as empty throughout, which is a useful signal:
the CR shows spec.layers populated and status.layerStatuses empty at the same time.

Steps to reproduce

1. Create a Lambda function without any layers (e.g. with Terraform, CloudFormation, or
   the console), so that GetFunction returns no Configuration.Layers.

2. Adopt it with the controller:

apiVersion: lambda.services.k8s.aws/v1alpha1
kind: Function
metadata:
  name: my-function
  annotations:
    services.k8s.aws/adoption-policy: adopt
    services.k8s.aws/adoption-fields: |
      {"name": "my-function"}
spec:
  name: my-function
  # ...

3. Once adopted, add a layer to the spec:

kubectl patch function my-function --type=merge \
  -p '{"spec":{"layers":["arn:aws:lambda:us-east-1:177933569100:layer:AWS-Parameters-and-Secrets-Lambda-Extension:88"]}}'

4. Observe the result:

# The CR has the layer in spec, but status is empty
kubectl get function my-function \
  -o jsonpath='spec.layers={.spec.layers}{"\n"}status.layerStatuses={.status.layerStatuses}{"\n"}'
# spec.layers=["arn:aws:lambda:us-east-1:177933569100:layer:AWS-Parameters-and-Secrets-Lambda-Extension:88"]
# status.layerStatuses=

# The condition reports success
kubectl get function my-function -o jsonpath='{.status.conditions[?(@.type=="ACK.ResourceSynced")].status}'
# True

# AWS still has no layers
aws lambda get-function-configuration --function-name my-function --query 'Layers'
# null

The controller logs show no Spec.Layers entry in any "desired resource state has changed" diff, and no error. In our environment, across the full controller log buffer,
diffs were emitted for Spec.Handler, Spec.MemorySize, Spec.Timeout, Spec.Environment,
Spec.Tags and Spec.Code — and never once for Spec.Layers.

CloudTrail confirms the field is absent from the API call. Both UpdateFunctionConfiguration
events issued by the controller in the same reconcile window carried other fields but no
layers key in requestParameters:

UpdateFunctionConfiguration  by <controller-role>
   handler : src/handler.handler
   layers  : <absent from requestParameters>

UpdateFunctionConfiguration  by <controller-role>
   memorySize : 512
   timeout    : 30
   layers     : <absent from requestParameters>

Expected outcome

Adding ARNs to spec.layers should attach those layers to the function, regardless of
whether the function currently has layers in AWS. Concretely, sdkFind should set
ko.Spec.Layers = nil in the else branch — mirroring what it already does for
ko.Status.LayerStatuses — so that latest reflects the actual AWS state and the delta can
be computed correctly.

This appears to be the same class of defect as #1413 (Spec.Timeout omitted from
UpdateFunctionConfiguration), which was fixed in lambda-controller#41.

Environment

• Kubernetes version: v1.34.9
• Using EKS (yes/no), if so version? yes, EKS 1.34
• AWS service targeted (S3, RDS, etc.): Lambda

Additional details:

• lambda-controller version: v1.5.8 (public.ecr.aws/aws-controllers-k8s/lambda-controller:1.5.8)
• Feature gates: ResourceAdoption: true, ReadOnlyResources: true
• The affected code is byte-identical in v1.5.8, v1.16.0 and main, so the bug is
  present in the latest release. Diffing the if resp.Configuration.Layers != nil { ... } else { ... } block across those three revisions produces no differences.
• Reproduced on three separate adopted functions in the same cluster; all three show
  spec.layers populated, status.layerStatuses empty and no layers in AWS.

[github-actions]

Hello @suprahh 👋 Thank you for opening an issue in ACK! A maintainer will triage this issue soon.

We encourage community contributions, so if you're interested in tackling this yourself or suggesting a solution, please check out our Contribution and Code of Conduct guidelines.

You can find more information about ACK on our website.

[suprahh]

I opened aws-controllers-k8s/lambda-controller#239 with a fix for this, and I
was able to reproduce and verify it end to end. Adding the details here in case
they help with the review.

Root cause

The relevant code is not generated by code-generator: it lives in
templates/hooks/function/sdk_read_one_post_set_output.go.tpl and is injected
into pkg/resource/function/sdk.go through the sdk_read_one_post_set_output
hook, so the fix is self-contained in lambda-controller.

sdkFind() builds the latest resource from a deep copy of the desired one.
When GetFunction returns no layer, the else branch only cleared
Status.LayerStatuses and left Spec.Layers holding the desired value
inherited from that deep copy, so desired and latest were identical and no
delta was ever computed.

It was introduced in #136, which added the else branch. The e2e test that
came with that same PR (test_function_layers) creates the function with
layers already attached and only replaces them, so this path was never covered.

Verification

Run against a real AWS account in us-west-2, with the controller running
locally against a kind cluster.

The new e2e test (test_function_add_layers_to_function_without_layers, added
in #239) starts from a function without layers and then adds them. Before the
fix:

tests/test_function.py::TestFunction::test_function_add_layers_to_function_without_layers FAILED
>       assert len(function["Configuration"]["Layers"]) == len(layers_list)
E       KeyError: 'Layers'

After the fix:

tests/test_function.py::TestFunction::test_function_add_layers_to_function_without_layers PASSED
1 passed, 19 deselected, 1 warning in 116.43s

test_function_layers still passes, so replacing existing layers is unaffected.

I also reproduced the original adoption scenario by hand. A Function adopted
with the adoption annotations, with no layers in AWS, then patched with
spec.layers:

LAYERS_SPEC=["arn:aws:lambda:us-west-2:017000801446:layer:AWSLambdaPowertoolsPythonV2:68"]
LAYER_STATUSES=
SYNCED=True

while AWS still reported "Layers": null. Restarting with the fixed controller,
without touching the CR, reconciled it on its own:

LAYERS_SPEC=["arn:aws:lambda:us-west-2:017000801446:layer:AWSLambdaPowertoolsPythonV2:68"]
LAYER_STATUSES=[{"arn":"arn:aws:lambda:us-west-2:017000801446:layer:AWSLambdaPowertoolsPythonV2:68","codeSize":4248334}]
SYNCED=True

and get-function-configuration then listed the layer.

Related

While verifying this I found a second bug on the same field, in the opposite
direction: emptying spec.layers never detaches the layers in AWS. It is
independent of this one and I am filing it separately.