Airbyte local installation on fresh linux installation fails with connection refused errors on temporal pod -- root cause is db connection

[ChrisR48]

Helm Chart Version

2.2.0

What step the error happened?

On deploy

Relevant information

Enviorment:

• Host: RHEL 9
• Kernel: 5.14.0-687.36.1.el9_8.x86_64
• Docker: 29.7.2
• abctl: v0.30.4

Executing local installation will lead to following error after some time:

ERROR   i.m.r.Micronaut(handleStartupException):362 - Error starting an of type [io.airbyte.commons.temporal.TemporalInitializationUtils]
          ...
          Message: Could not create Temporal client within max timeout!
          ...
          Caused by: java.lang.RuntimeException: Could not create Temporal client within

Checking the kubernet cluster shows several pods crashing and restarting, mainly because temporal is not reachable. My knowledge of kubernet and the airbyte setup gets a bit weak here, so hopefully i get all information right.

Within the temporal pod the log-output is:

nc: bad address 'airbyte-db-svc.airbyte-abctl.svc.cluster.local' Waiting for PostgreSQL to startup.

But the database pod is running and accepts connection.

Some test (with the help of AI) shows that the temporal pod is unable to resolve the address of the db pod.

getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local → exit code 2

Comparing the dns setup of the temporal pod with the manifest server shows no differences.

Testing the dns resolver with a new temporal pod and an alpine image shows the same issue.

kubectl run temporal-dns-test -n airbyte-abctl --image=temporalio/auto-setup:1.27.2 --restart=Never --command -- sleep 300

kubectl exec -n airbyte-abctl temporal-dns-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local
-> Failed
`

kubectl run alpine-test -n airbyte-abctl --image=alpine:3.21 --restart=Never --command -- sleep 300

kubectl exec -n airbyte-abctl alpine-test -- getent hosts airbyte-db-svc.airbyte-abctl.svc.cluster.local
-> Failed

At the end I was able to start airbyte with the following patch which change the alpine based pods dns ndots setting to "1" :

kubectl patch deployment airbyte-abctl-temporal -n airbyte-abctl --type='strategic' -p '{
    "spec": {
      "template": {
        "spec": {
          "dnsConfig": {
            "options": [
              {
                "name": "ndots",
                "value": "1"
              }
            ]
          }
        }
      }
    }
  }'

Directly after this the other pods were able to start successful.

I have understood that this is more like a temporary workaround which will be lost the next time I use abctl local install.

For now I would like to share my experience so that it can be solved for others as well.

If there is an easy fix for it, maybe it would help to be included in the installation guide.

Relevant log output

---

Internal Tracking: https://github.com/airbytehq/oncall/issues/13313

[octavia-bot]

AI Triage Starting...

Thank you for opening this issue! I'm an AI assistant that will do an initial review and may:

• Ask clarifying questions if needed
• Suggest relevant documentation or workarounds
• Flag this for internal visibility

Note: Community issues are reviewed on a best-effort basis and do not have a guaranteed response time. For additional help:

• Join the Airbyte Community Slack to connect with other users and community members
• If you're an Airbyte Cloud customer, you can open a support ticket or contact your account representative, referencing this issue URL

View playbook

• 🔗 Session URL: https://app.devin.ai/sessions/a73e6b3f1a6f4a47ba673c61a0d936b1

✅ Devin AI session created successfully!

[airbyte-support-bot]

Hi Christian Rose (@ChrisR48), thank you for reporting this, and for the detailed writeup of your environment and investigation — the ndots finding is very useful context.

We're looking into this and have escalated it to our team for investigation: airbytehq/oncall#13313

A few clarifications would help:

• Could you share the contents of /etc/resolv.conf on the RHEL 9 host (particularly the search list and any options line)?
• Which resolver stack is in use on the host — systemd-resolved, NetworkManager, or something else?
• Did abctl local install complete end-to-end after patching only the Temporal deployment, or did other Alpine-based deployments need the same patch?

If you share any further logs or config, please mask or remove any sensitive information (API keys, passwords, tokens, connection strings) first. If any evidence currently lives behind a link, please paste the relevant excerpt inline in a fenced code block.

---

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.

[ChrisR48]

Hi,

here are the requested informations:

cat /etc/resolv.conf
# Generated by NetworkManager
search xxxxxxxxxxxxxxx.de
nameserver 192.168.178.254
nameserver 2003:d6:b713:7cfe:82ee:73ff:feea:c056

Used resolver stack: NetworkManager

abctl local install was still running while I did the investigation an patching. Directly after the patch (and some time for the pods to restart) the installation completed. There was no need to patch any other pods beside temporal.

[ChrisR48]

One small addon. Same error occurs on a fresh installed Ubuntu 22.04 server.