Archive sooner - #17
Conversation
…need to wait for daily snapshots.
|
|
||
| echo "Deleting ${INDEX_NAME} from Elasticsearch." | ||
| curl -w "\n" -sS -XDELETE ${INDEX_URL} | ||
| echo curl -w "\n" -sS -XDELETE ${INDEX_URL} |
There was a problem hiding this comment.
Is this change intentional? (and what is trying to achieve?)
There was a problem hiding this comment.
nope, this is ... probably why nothing is happening lol. i left debug echoes.
| { | ||
| : ${1:?"Error: expected index name passed as parameter"} | ||
| local INDEX_NAME=$1 | ||
| local SNAPSHOT_URL=${REPOSITORY_URL}/${INDEX_NAME} |
There was a problem hiding this comment.
You can probably skip SNAPSHOT_URL: it never gets called in this function.
|
|
||
| CUTOFF_DATE=$(date --date="${MAX_DAYS_TO_KEEP} days ago" +"%Y.%m.%d") | ||
| echo "$(now) Archiving all indexes with logs before ${CUTOFF_DATE}." | ||
| ARCHIVE_CUTOFF_DATE=$(date --date="1 days ago" +"%Y.%m.%d") # why would we wait weeks to archive? |
There was a problem hiding this comment.
I get where you're coming from but you can probably skip that comment here :)
There was a problem hiding this comment.
More importantly: I'm a little concerned about using exactly "1 day" as the cutoff. Using exactly one day sort of assumes that everybody agrees on what the date is — i.e. that they use the same timezone and their time is in sync —, which may not always be 100% accurate.
Unfortunately, as I noted in this other PR, snapshotting too early in this script means you may not end up snapshotting everything...
I figure we could either create a final snapshot before deleting an index, or give it 2 days before snapshotting..?
There was a problem hiding this comment.
Yeah, 2 days is good, but I'm unclear how the container running this script and the container the ES data is in would disagree.
My main concern is having ES die, and losing 7, 14, 30, whatever days of data because the archive waits until delete.
There was a problem hiding this comment.
Yeah, 2 days is good, but I'm unclear how the container running this script and the container the ES data is in would disagree.
If you're assuming this is running on Enclave, then yes, this is unlikely to happen since we're running Logstash using UTC as the timezone, and you're using this container which also uses UTC as the timezone.
However, that might not be the case for everyone using this: if someone runs Logstash in UTC, but executes this script from a different TZ, they might end up backing up their logs too early because the script thinks today is day N+1 (and thus backs up the logstash-N index), but Logstash thinks today is day N and continues to write to the logstash-N index, thus resulting in any logs written after the backup script runs to never be archived.
My main concern is having ES die, and losing 7, 14, 30, whatever days of data because the archive waits until delete.
Worth mentioning that, if you're assuming this is running on Enclave, this is largely moot considering the ES disk is actually backed up on a daily basis, though I can understand that you wanting to have your own archives.
| else | ||
| echo "$(now): ${index_name} archival failed." | ||
| fi | ||
| if [[ "${index_name:9}" > "${DELETE_CUTOFF_DATA}" ]]; then |
There was a problem hiding this comment.
I'm fairly certain this should be DELETE_CUTOFF_DATE, not DELETE_CUTOFF_DATA (or this will delete all indices that are archive-able).
TBH, it's a real problem that this script does not run with set -o nounset, but for now, that's what it is :/
|
FWIW I find the inconsistent nomenclature ( |
| else | ||
| echo "$(now): ${index_name} archival failed." | ||
| fi | ||
| if [[ "${index_name:9}" > "${DELETE_CUTOFF_DATA}" ]]; then |
There was a problem hiding this comment.
Should the deletion be performed in all cases, even if the archival failed?
3 participants
Still testing this, but one of the concerns I had with the default behavior was waiting to archive the data until it was time to delete. If the data is already segmented by day, anything older than a day should always be archived.