Automate cluster setup

[robertgartman]

This PR will (in listed order):

1. Ensure that all CouchDB containers get all peers registered in _node db
2. Do the POST http://127.0.0.1:5984/_cluster_setup , payload {"action": "finish_cluster"} on pod with ordinal nr 0 (if username and password provided)

The intention is that the cluster will succeed with setup even if some of the CouchDB nodes are restarted during cluster formation (e.g. du to Liveness or Readiness settings).

It's my first hands-on with Python so... pick, chose and modify as you see fit if you chose to accept this PR.

Below are example logs from formation of a 3-node cluster.

kubectl logs -f my-release-couchdb-2 -c couchdb-statefulset-assembler       
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
	| Got the following peers' fqdm from DNS lookup: ['my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local']
Adding CouchDB cluster node my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Adding CouchDB cluster node my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Adding CouchDB cluster node my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Probing my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Probing my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Probing my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Cluster membership populated!
Intentionally skipping the POST to http://127.0.0.1:5984/_cluster_setup {"action": "finish_cluster"}

kubectl logs -f my-release-couchdb-1 -c couchdb-statefulset-assembler
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
	| Got the following peers' fqdm from DNS lookup: ['my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local']
Adding CouchDB cluster node my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Adding CouchDB cluster node my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Adding CouchDB cluster node my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Probing my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Probing my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Probing my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Cluster membership populated!
Intentionally skipping the POST to http://127.0.0.1:5984/_cluster_setup {"action": "finish_cluster"}

kubectl logs -f my-release-couchdb-0 -c couchdb-statefulset-assembler
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
	| Got the following peers' fqdm from DNS lookup: ['my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local']
Adding CouchDB cluster node my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local
	| Response: 201 {'ok': True, 'id': 'couchdb@my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local', 'rev': '1-967a00dff5e02add41819138abb3284d'}
Adding CouchDB cluster node my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local
	| Response: 201 {'ok': True, 'id': 'couchdb@my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local', 'rev': '1-967a00dff5e02add41819138abb3284d'}
Adding CouchDB cluster node my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Probing my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local for cluster membership
	| Cluster members in my-release-couchdb-0 not yet present in my-release-couchdb-2: {'couchdb@my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local', 'couchdb@my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local'}
Probing my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local for cluster membership
	| Cluster members in my-release-couchdb-0 not yet present in my-release-couchdb-1: {'couchdb@my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local', 'couchdb@my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local'}
Probing my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Resolving SRV record _couchdb._tcp.my-release-couchdb.default.svc.cluster.local
	| Got the following peers' fqdm from DNS lookup: ['my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local', 'my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local']
Adding CouchDB cluster node my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Adding CouchDB cluster node my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Adding CouchDB cluster node my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local to this pod's CouchDB.
	| Request: PUT http://127.0.0.1:5986/_nodes/couchdb@my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local
	| Response: 409 {'error': 'conflict', 'reason': 'Document update conflict.'}
Probing my-release-couchdb-2.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Probing my-release-couchdb-1.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Probing my-release-couchdb-0.my-release-couchdb.default.svc.cluster.local for cluster membership
	| In sync!
Cluster membership populated!
Get the cluster up and running
	| Request: POST http://127.0.0.1:5984/_cluster_setup , payload {"action": "finish_cluster"}
	|	Response: 201 {'ok': True}
	| Sweet! Just a final check for the logs...
	| Request: GET http://127.0.0.1:5984/_cluster_setup
	|	Response: 200 {'state': 'cluster_finished'}
Time to relax!