Save batch logs locally first, then sync

[IgnacioHeredia]

We should save first locally because (1) we avoid error in networks hiccups, (2) we can save intermediate steps, to be able to show logs before the command finishes. Otherwise rclone wouldn't sync until the file is closed (i.e. script has completed).

This is an first try but I can't make it work for some reason and I don't have more time to debug, so I'm leaving this here for my future self.

 usertask["Config"]["command"] = "/bin/bash"
    usertask["Config"]["args"] = [
        "-c",
        (   "export LOCAL_DIR=/tmp/ai4os-batch/${NOMAD_JOB_ID} && "
            "export REMOTE_DIR=/storage/ai4os-storage/batch/${NOMAD_JOB_ID} && "
            "mkdir -p $LOCAL_DIR $REMOTE_DIR && "
            "(while true; do cp -f $LOCAL_DIR/log.txt $REMOTE_DIR/log.txt 2>/dev/null || true; sleep 20; done) & "
            "SYNC_PID=$! && "
            "/bin/bash /srv/user-batch-commands.sh >> $LOCAL_DIR/log.txt 2>&1; "
            "EXIT_CODE=$?; cp -f $LOCAL_DIR/log.txt $REMOTE_DIR/log.txt 2>/dev/null || true; "
            "kill $SYNC_PID 2>/dev/null || true; wait $SYNC_PID 2>/dev/null || true; exit $EXIT_CODE"
        ),
    ]