Backup Implementation

AWS Glacier Deep Archive

Here we give an overview of the key steps for setting up backups to AWS Glacier Deep Archive, focusing on the use of Globus. For in-depth instructions for each step, including screenshots, please see this Google doc providing informal documentation.

• If needed, create an AWS account with billing to a UC Berkeley chartstring.
• Create an S3 bucket in which backups will be placed.
  • Create a lifecycle policy for the bucket that transitions objects to the Glacier Deep Archive storage class after a minimal period of time.
  • Decide whether to set up versioning in the bucket, such that files changed in the local storage will be uploaded as new versions, with the old versions retained. Given the focus is likely to be on backing up data, we anticipate this may not be widely-used, particularly given that new versions may be explicitly created in separate directories in the local storage as a means of managing local versions.
• Create an AWS IAM (identity and access management) account. This is necessary for Globus to transfer files to AWS S3.
• Transfer your files to AWS. 
  • If using Globus (either the web interface or Globus CLI), use the IAM credentials for AWS and the collection named “UCB AWS S3 Collections”.
  • Decide whether to sync files that already exist on the AWS side, thereby only transferring new or changed files.
  • You can also use AWS-cli or rclone to transfer files.
• Alternatively, set up the Globus timer service to automate your transfers.

Wasabi

Here we give an overview of the key steps for setting up backups to Wasabi, focusing on the use of Globus. For in-depth instructions for each step, including screenshots, please see this Google doc providing informal documentation.

• If needed, create a Wasabi account with billing to a UC Berkeley chartstring.
  • As part of setting up your account, you will generate an access key ID and secret access key that serve as your Wasabi credentials.
• Create a bucket on Wasabi in which backups will be placed.
  • Decide whether to set up versioning in the bucket, such that files changed in the local storage will be uploaded as new versions, with the old versions retained. Given the focus is likely to be on backing up data, we anticipate this may not be widely-used, particularly given that new versions may be explicitly created in separate directories in the local storage as a means of managing local versions.
• Transfer your files to Wasabi.
  • If using Globus (either the web interface or Globus CLI), use the collection named "UCB S3 Wasabi Data".
  • If it is the first time using Globus to access Wasabi you will need to consent to allowing Globus to access your Wasabi account, based on your Wasabi credentials.
  • Decide whether to sync files that already exist on the Wasabi side, thereby only transferring new or changed files.
  • You can also use AWS-cli or rclone to transfer files.
• Alternatively, set up the Globus timer service to automate your transfers.