Data Transfer Overview

This is an overview of how to transfer data to or from the Berkeley Savio and Vector clusters.

When transferring data using file transfer software, you should connect only to the cluster's Data Transfer Node, dtn.brc.berkeley.edu. (Note: if you're using Globus, you'll instead connect to Savio's Globus endpoint ucb#brc)

After connecting to the Data Transfer Node, you can transfer files directly into (and/or copy files directly from) your Home directory, Group directory (if applicable), and Scratch directory.

Executables located in scratch cannot be run on the DTN

The scratch directory is mounted noexec on the DTN for security reasons, which means that you cannot run executables in scratch while on the DTN (you'll receive a 'Permission denied' error). So you'll need to place any of your own executables used for downloading data in your home or group directory.

For information on making your files accessible to other users (in particular members of your group), see these instructions.

Medium- to large-sized data transfers

When transferring a large number of files and/or large files, we recommend you use:

  • Globus Connect: This method allows you to make unattended transfers that are fast and reliable. For basic instructions, see Using Globus Connect.

You can additionally use GridFTP or BBCP for this purpose ...

Small-sized data transfers

When transferring a modest number of smaller-sized files, you can also use:

You can additionally use protocols like FTPS and tools like Rsync for this purpose ...

Transfers to/from repositories under version control

When your code and/or data are stored in repositories under version control, client software is available for accessing them via:

  • Git
  • Mercurial
  • Subversion (SVN)

See Accessing Software for information on finding and loading this software via the BRC supercluster's Environment Modules.

Transfers to/from specific systems

For bDrive (Google Drive) and Box, we recommend using rclone to transfer data to and from Savio. 

  • See here for instructions for using Box and bDrive with Savio.
  • See here for additional information about bDrive (Google Drive) provided by our Research Data Management program.

Additional tutorials for transferring files to/from Amazon Web Services (AWS) S3 (for instance, see the examples of making an AWS S3 rclone configuration here and here, as well as using AWS CLI to transfer files to and from AWS S3) and other popular data storage systems are in planning or development. If you have any interest in working on or testing one of these, or have suggestions for other data transfer tutorials, please contact us via our Getting Help email address!