Service offline after container auto update

[Tom7320]

Hi there,

sometimes after a container auto update the Tailscale service is still there but offline even though the container is running and perfectly fine. I then have to restart Docktail to fix it. Is it possible to somehow "harden" the mechanism Docktail checks whether the service is online? The logs don't show anything special AFAICS. Thanks a lot!

Regards

Thorsten

[Paddy132a]

I have been seeing what appears to be a related issue when creating new Tailscale Services via Docktail. The Docker container is healthy, tailscale serve status --json shows the expected service configuration, and Docktail reports a successful reconciliation. However, the new Service often appears as Offline (or 0 hosts) in the Tailscale Services UI and is unreachable. Re-running the Docktail reconciliation sometimes fixes it, but often it does not. In many cases the only solution is to wait several hours (sometimes overnight), after which the Service suddenly appears online without any configuration changes. Existing Services remain unaffected; this seems to affect only newly created Services. Has anyone else observed this behaviour, or is there a known delay or race condition between Docktail reconciliation and Tailscale Services becoming active?

ps very happy with Docktail generally - over 100 Tailscale Services created and working well!!

[Tom7320]

It would be nice if Docktail had a kind of verification mechanism to check the online status of all services on a regular basis. But unfortunately I do not know anything about the internal workings of Docktail in order to determine whether that is possible...

[flammable]

I just started using DockTail, and have observed the same thing. I moved a bunch of services from my reverse proxy to Tailscale services over the weekend.

When I add a new service, it shows as "0 hosts" in Tailscale, though I can see the container is running (and is reachable via my reverse proxy). When I add another service, it also shows as "0 hosts", but the first service suddenly shows 1 host and works properly.

I don't know if this is because my services are using port 443 and HTTPS or not. I don't think I saw this issue when testing with HTTP and port 80. Restarting DockTail or the service's Docker container doesn't always fix it for me, but adding a new service consistently fixes the previous one.

That said, this is a fantastic tool, and I'm very appreciative that it exists. Many thanks to @marvinvr for making this available. I've wanted to move services behind VPN for a long time, and I'm so happy to finally have completed that. It was so easy, too! 😄

[marvinvr]

I've seen this happen a few months ago and the problem for me was actually related to the Tailscale mount becoming stale between the container and the host system. I at that point updated all the docs (i hope, tell me if I didn't) to mount '/var/run/tailscale' into the container instead of '/var/run/tailscale/tailscaled.sock'. Can you confirm how you set it up in the compose and if not try mounting the folder instead and see if it resolves the issue?

[flammable]

I think that's right - the docs guided me towards mapping /var/run/tailscale:/var/run/tailscale. After I got everything working and created all of the services I'm going to make for now, I added the Docker socket proxy, and things have worked pretty consistently since then.

I don't know if it matters, but I'm using Unraid 7.3.2 with the native Tailscale plugin.

[Tom7320]

Unraid 7.3.2 on my side. Last night auto update of Stirling PDF:

I mount /var/run/tailscale since the beginning:

Restart of Docktail fixes the problem.

[GreymatterPrison]

I am seeing the same issue on every update. A restart of the stack brings the service back online for a few seconds then they are offline again. The only fix for me is to redeploy the stack after each update.

Is there a way to mitigate this?

[Paddy132a]

EXPERIMENT

I’ve now reproduced this in a controlled test, including the behaviour @flammable reported.

1. host-a has 6 existing DockTail-managed Services. A disposable HTTPS/443 nginx Service came online normally and remained reachable.

2. host-b has 97 existing DockTail-managed Services and acts as a broad Service host across the tailnet, serving applications used from multiple tailnet locations. It uses the correct whole-directory /var/run/tailscale mount, so the stale socket-file issue is ruled out.

On host-b, in sequence:

• Create A → DockTail reconciles successfully; A is in Serve config and AdvertiseServices, but shows 0 hosts and is unreachable.
• Create B → A becomes online and HTTP 200; B becomes the new unreachable Service. - the @flammable effect
• Create C → B advances from TCP-unreachable to TCP-reachable, but TLS still times out; C remains TCP-unreachable.

All backends and DockTail remained healthy. No restart, manual reconciliation or corrective action was performed.

Current preserved state:

• A: configured + advertised + HTTP 200
• B: configured + advertised + TCP works, TLS/HTTP fails
• C: configured + advertised + TCP fails

So Serve config present + AdvertiseServices present is not sufficient for an actually usable Service, and creating another Service appears to advance the previous Service’s state.

I have timestamped Docker/DockTail/Serve/prefs and independent reachability evidence for the sequence.

I can run the PR #76 diagnostics, but that requires replacing/restarting DockTail on host-b and may disturb the preserved broken state.

I shall proceed to PR #76 as phase B - should I capture anything else be n this current phase?

[Paddy132a]

EXPERIMENT CONTINUED
Follow-up: overnight recovery

A further update on the A/B/C experiment above.

I left the broken Services completely untouched and ran an independent passive watcher from another tailnet host.

The delayed recovery was much longer than the initial observation window suggested:

• B was created at 07:51:05Z.

  • It first accepted TCP after C was created.
  • It did not reach valid TLS / HTTP 200 until 10:26:01Z.
  • That was 2h34m56s after B was created and 2h19m23s after C was created.

• C was created at 08:06:38Z.

  • It was still TCP-unreachable more than two hours later.
  • It first accepted TCP at 16:30:11Z.
  • It reached valid TLS / HTTP 200 at 16:30:41Z.
  • That was 8h24m03s after C was created.

No Docker, DockTail or tailscaled restart occurred, no manual reconciliation or Serve command was issued, and no corrective action was taken.

So this reproduces the behaviour I originally described: a newly created Service can remain locally configured and advertised but unusable for hours — in C's case effectively overnight — and then become fully operational without intervention.

I also checked all eight of my DockTail hosts. This problem appears only on host-b.

Before this investigation, host-b managed 96 DockTail Services. The other seven hosts manage:

• 25
• 9
• 9
• 7
• 6
• 1
• 1

So host-b is a substantial scale outlier, although I am not suggesting that Service count is proven to be the cause.

One particularly useful comparator is another Ubuntu/x86_64 host running Docker 29.6.2, Tailscale 1.102.2, host networking, the same whole-directory /var/run/tailscale mount and RECONCILE_INTERVAL=600s. It has only 9 DockTail-managed Services and behaves normally.

I then moved on to testing PR #76 separately.

[Paddy132a]

EXPERIMENT CONTINUED AND CONCLUDED (FOR NOW)

PR #76 follow-up: reproduction under diagnostics

Thanks for PR #76. I built and ran exact commit 3e436c99f7b21c22d6b9e7c698a5db57376b5638 on host-b with the JSONL diagnostics enabled.

The same pattern reproduced again.

• A became locally configured + advertised but remained externally unusable for more than 50 minutes and five 600-second reconciliation cycles.
• Creating B coincided with A becoming usable while B became the new broken Service.
• Creating C coincided with B becoming usable while C became broken.
• Creating D coincided with C becoming usable while D became broken.

There is one timing caveat on A/B: the watcher first saw A accept TCP 8 seconds before B was created, so I do not claim B caused that particular TCP transition.

The C and D results were cleaner: the previously broken Service became usable within seconds of creating the next Service, well away from the 600-second periodic reconciliation cadence.

What the PR #76 diagnostics showed

For A/B/C/D:

• the nginx backend remained healthy
• the Service was locally configured
• the Service was locally advertised
• no per-Service diagnostic anomaly was reported
• daemon health was essentially clean apart from brief ACME certificate-fetch messages

When the preceding broken Service became externally usable:

• its configured state did not change
• its advertised state did not change
• its backend destination did not change
• no corresponding local anomaly appeared

The VIP fingerprint changed when the new Service was added, not when the previous Service later became externally usable.

So the significant state transition is visible from the external data plane, but not in the local Serve / AdvertiseServices state currently recorded by PR #76.

D is currently preserved in the broken state:

• configured
• advertised
• healthy backend
• no PR76 anomaly
• externally TCP-unreachable

I have left it untouched in case there is anything else you would like captured.

I have timestamped PR76 JSONL plus independent DNS / TCP / TLS / HTTP observer evidence for the full sequence.

For transparency, I used ChatGPT + Codex together to organise this investigation: ChatGPT structured the experimental gates and analysis, while Codex performed the guarded deployments, host audits and timestamped evidence collection. The observations themselves come directly from DockTail/PR76, Tailscale state and independent external probes.

[Paddy132a]

Experiment follow-up — delayed / overnight recovery

A further update on the original A/B/C experiment.

I left the broken Services completely untouched and ran an independent passive watcher from another tailnet host.

Delayed recovery

• B was created at 07:51:05Z.

  • It first accepted TCP after C was created.
  • It did not reach valid TLS / HTTP 200 until 10:26:01Z.
  • That was 2h34m56s after B was created and 2h19m23s after C was created.

• C was created at 08:06:38Z.

  • It was still TCP-unreachable more than two hours later.
  • It first accepted TCP at 16:30:11Z.
  • It reached valid TLS / HTTP 200 at 16:30:41Z.
  • That was 8h24m03s after creation — effectively an overnight recovery.

No Docker, DockTail or tailscaled restart occurred, no manual reconciliation or Serve command was issued, and no corrective action was taken.

So this reproduces the behaviour I originally described: a newly-created Service can be locally configured and advertised but unusable for hours, then become fully operational without direct intervention.

Cross-host comparison

I also checked all eight of my DockTail hosts. host-b is the only host where I can reproduce this.

Before this investigation, host-b managed 96 DockTail Services. The other seven hosts manage:

25, 9, 9, 7, 6, 1, 1

So host-b is a substantial scale outlier, although I am not suggesting Service count is proven to be the cause.

A useful negative comparator is another Ubuntu/x86_64 host running:

• Docker 29.6.2
• Tailscale 1.102.2
• host networking
• the same whole-directory /var/run/tailscale mount
• RECONCILE_INTERVAL=600s

That host has only 9 DockTail-managed Services and behaves normally.

I then moved on to testing PR #76 separately.