🔗 Synced from upstream: NOAA-GSL/zyra#288
This issue is mirrored from the upstream repository. Status changes here will be synced back.
Summary
zyra acquire ftp crashes the entire pipeline on any transient FTP hiccup. The FTP backend opens one connection and never reconnects or retries, so a single dropped socket mid-sync — which we see routinely against NOAA’s public FTP under load — fails the run with no recovery.
Where
src/zyra/connectors/backends/ftp.py:
• sync_directory() issues a per-file MDTM probe before each transfer. Over a large directory this is a burst of commands (“MDTM storm”) that can itself provoke the server into dropping the control connection (e.g. 421 Too many connections / idle-timeout / reset).
• ensure_ftp_connection() establishes the connection once but does not re-establish it if it has dropped — there’s no liveness check or reconnect.
• The per-file RETR/MDTM calls are not wrapped in retry/except, so the first ftplib/socket error (EOFError, socket.timeout, ConnectionResetError [Errno 104], error_temp 421) propagates straight out and aborts acquire.
• The socket timeout (≈30 s) makes a stalled transfer fail hard rather than recover.
Impact
Scheduled, automated pipelines that sync incrementally from NOAA FTP fail intermittently for reasons entirely outside the pipeline’s control. The data is fine — a re-run usually succeeds — so these are false-positive failures. Downstream consumers have to paper over it with whole-pipeline retries.
Suggested fixes (any subset helps)
1. Reconnect-on-failure in ensure_ftp_connection() — detect a dead control connection (failed NOOP/PWD) and transparently re-establish before the next command.
2. Bounded per-file retry with backoff around RETR (and the MDTM probe) — reconnect + retry a few times before giving up.
3. Reduce the MDTM storm — make the per-file MDTM probe optional (e.g. skip when size+name already match, or use MLSD/a single listing for mtimes instead of one MDTM per file).
4. Optional FTP keepalive / connection reuse tuning to avoid idle-timeout drops on long syncs.
Repro sketch
Run acquire ftp with --sync-dir over a directory of a few thousand files against a busy FTP server; observe an intermittent ftplib.error_temp/ConnectionResetError from the MDTM/RETR path with no reconnect attempt.
Environment
Observed via the pinned ghcr.io/noaa-gsl/zyra-scheduler image in automated GitHub Actions runs against NOAA public FTP.
Summary
zyra acquire ftp crashes the entire pipeline on any transient FTP hiccup. The FTP backend opens one connection and never reconnects or retries, so a single dropped socket mid-sync — which we see routinely against NOAA’s public FTP under load — fails the run with no recovery.
Where
src/zyra/connectors/backends/ftp.py:
• sync_directory() issues a per-file MDTM probe before each transfer. Over a large directory this is a burst of commands (“MDTM storm”) that can itself provoke the server into dropping the control connection (e.g. 421 Too many connections / idle-timeout / reset).
• ensure_ftp_connection() establishes the connection once but does not re-establish it if it has dropped — there’s no liveness check or reconnect.
• The per-file RETR/MDTM calls are not wrapped in retry/except, so the first ftplib/socket error (EOFError, socket.timeout, ConnectionResetError [Errno 104], error_temp 421) propagates straight out and aborts acquire.
• The socket timeout (≈30 s) makes a stalled transfer fail hard rather than recover.
Impact
Scheduled, automated pipelines that sync incrementally from NOAA FTP fail intermittently for reasons entirely outside the pipeline’s control. The data is fine — a re-run usually succeeds — so these are false-positive failures. Downstream consumers have to paper over it with whole-pipeline retries.
Suggested fixes (any subset helps)
1. Reconnect-on-failure in ensure_ftp_connection() — detect a dead control connection (failed NOOP/PWD) and transparently re-establish before the next command.
2. Bounded per-file retry with backoff around RETR (and the MDTM probe) — reconnect + retry a few times before giving up.
3. Reduce the MDTM storm — make the per-file MDTM probe optional (e.g. skip when size+name already match, or use MLSD/a single listing for mtimes instead of one MDTM per file).
4. Optional FTP keepalive / connection reuse tuning to avoid idle-timeout drops on long syncs.
Repro sketch
Run acquire ftp with --sync-dir over a directory of a few thousand files against a busy FTP server; observe an intermittent ftplib.error_temp/ConnectionResetError from the MDTM/RETR path with no reconnect attempt.
Environment
Observed via the pinned ghcr.io/noaa-gsl/zyra-scheduler image in automated GitHub Actions runs against NOAA public FTP.