rclone integration

[lpeabody]

Rclone (https://rclone.org/) has been a really useful tool for migrating files from other platforms to Pantheon. It's not necessarily Pantheon specific so it may be more appropriate to put this in Polymer core. In a nutshell though we need to be able to generate rclone migration scripts that can transfer files from source to destination. Each file migration script should migrate a complete set of files.

• generate rclone:
  • configuration file
  • transfer script
  • audit script
• By default, we'll assume we have a source that has public and private files on-server, and the files need to be transferred to a Pantheon environment in Pantheon's standard public/private file system setup.
• There should be an rclone script that audits the source file systems for Pantheon compatibility checks. https://docs.pantheon.io/guides/filesystem/large-files
  • Files over 256MB in size cannot be hosted on Pantheon (flag these as files that require migration to an external file system)
  • Files over 100MB in size cannot be uploaded through Drupal (these files should also be migrated to external file system, and fields they were sourced from need to be configured to use the external file system)
  • Files <= 100MB in size can be uploaded through Drupal fine

example transfer script:

#!/usr/bin/env bash

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
export RCLONE_CONFIG="${SCRIPT_DIR}/pantheon-source-to-dest.conf"

rclone sync \
  source_prod:/mnt/gfs/source/sites/site/files \
  pantheon_dev:/files \
  --exclude-from "${SCRIPT_DIR}/excludes.txt" \
  --exclude private/ \
  --size-only \
  --progress \
  --verbose \
  --transfers=32;

rclone sync \
  source_prod:/mnt/files/source.prod/sites/site/files-private \
  pantheon_dev:/files/private \
  --exclude-from "${SCRIPT_DIR}/excludes.txt" \
  --size-only \
  --progress \
  --verbose \
  --transfers=32;