Retry transient connection errors on idempotent iRODS reads

[pauldg]

The iRODS object store's idempotent read operations (_exists_remotely, _get_remote_size, _download) don't retry on potentially transient connection errors and fails the job. In my set up these show up as NetworkException (from an idle pooled connection reset by a firewall/proxy) and ssl.SSLError (a dropped TLS handshake on a fresh connection); both of these resolve on a new connection.

This PR is a small decorator that retries a set number of times with a short backoff. Writes, deletes, not found (DataObjectDoesNotExist/CollectionDoesNotExist) are left untouched.

Transient errors like these are unavoidable on networked storage. The S3 backends already have a similar mechanism:

• built-in num_download_attempts param:

  galaxy/lib/galaxy/objectstore/s3_boto3.py

  Line 180 in 1e9cd1d

  "num_download_attempts": int,

• a looped try-catch like this PR:

  galaxy/lib/galaxy/objectstore/s3.py

  Lines 251 to 273 in 1e9cd1d

  	def _get_bucket(self, bucket_name):
  	"""Sometimes a handle to a bucket is not established right away so try
  	it a few times. Raise error is connection is not established."""
  	last_error = None
  	for i in range(5):
  	try:
  	bucket = self.conn.get_bucket(bucket_name)
  	log.debug("Using cloud object store with bucket '%s'", bucket.name)
  	return bucket
  	except S3ResponseError as e:
  	last_error = e
  	try:
  	log.debug("Bucket not found, creating s3 bucket with handle '%s'", bucket_name)
  	self.conn.create_bucket(bucket_name)
  	except S3ResponseError:
  	log.exception("Could not get bucket '%s', attempt %s/5", bucket_name, i + 1)
  	time.sleep(2)
  	# All the attempts have been exhausted and connection was not established,
  	# raise error
  	if last_error:
  	raise last_error
  	else:
  	raise Exception("Failed to connect to target object store.")

For the python-irodsclient there is no equivalent, I'll also submit an issue and/or PR to the python-irodsclient codebase to handle it there too.

Unit tests cover: recover-on-NetworkException, recover-on-ssl.SSLError, give-up after the max attempts, and no retry on success.

How to test the changes?

• I've included appropriate automated tests.

License

• I agree to license these contributions under the project's existing license.