Add support for streaming cli output?

[zstumgoren]

Refactor the S3 wrapper class to return real-time/streaming output from the CLI process rather than returning all output at tail of end processing. Will need to switch from subprocess.check_output to a custom use of Popen, likely with polling etc.

[zstumgoren]

fwiw, simply printing stdout at tail end of subprocess call appears to work fine. Streaming output would be most useful to avoid excessive memory usage and delayed CLI feedback when uploading very large files and/or a large number of files (on the order of hundreds or thousands).

[zstumgoren]

We agreed to hold off on implementation for now since it's a "nice-to-have". Bumping to 0.3.0 release and adding a Help Wanted label, in case someone else out there decides to poke at this.

[myersjustinc]

Started to refresh my memory on Popen and such, and got at least some success with the following:

proc = subprocess.Popen(
    ['./long.sh'], stdout=sys.stdout, stderr=subprocess.STDOUT,
    encoding='utf-8')
proc.wait()
if proc.returncode != 0:
    print('\n*** Error ***\n')

where long.sh is a bash script with an assortment of echo and sleep commands.

Doesn't seem ideal, though, and doesn't play as nicely with datakit_data.s3's logger.

The "right" way to do it probably is going to involve asyncio or something similar, but wanted to make note of what I've looked into so far.

[zstumgoren]

+1 Yep, I had kicked around some Popen implementations as well on first pass, but punted until we had time to vet solutions more thoroughly. We can include asyncio as a possible solution as well.