Connection refused, but actually not?

[BNolet]

    kind: pipeline
    name: default

    steps:
      - name: scp files
        image: appleboy/drone-scp
        settings:
          host: example.com
          username: brandon
          key:
            from_secret: SSH_KEY
          port:
            from_secret: PORT
          target: /home/user/directory/
          source: .

The pipeline scp files step seems to say that the connections to the host were refused but the evidence seems to suggest otherwise as the auth.log file reports every one of the connections made at the same time as accepted and successfully opened then closed. As well, sometimes the commands do execute and the files are indeed transferred. But when the connection is said to be refused 2 out of 4 times, the file will not get removed.

Either way, if any connection is "refused" (but not actually) the pipeline fails to complete.