This is an overview of how to transfer data to or from the Berkeley Research Computing (BRC) supercluster, consisting of the Savio and Vector high-performance computing clusters, at the University of California, Berkeley.

When transferring data using file transfer software, you should connect only to the supercluster's Data Transfer Node, dtn.brc.berkeley.edu. (Note: if you're using Globus Connect, you'll instead connect to the Globus endpoint ucb#brc)

After connecting to the Data Transfer Node, you can transfer files directly into (and/or copy files directly from) your Home directory, Group directory (if applicable), and Scratch directory.

For information on making your files accessible to other users (in particular members of your group), see these instructions.

Medium- to large-sized data transfers

When transferring a large number of files and/or large files, we recommend you use:

  • Globus Connect (formerly Globus Online): This method allows you to make unattended transfers which are fast and reliable. For basic instructions, see Using Globus Connect.

You can additionally use GridFTP or BBCP for this purpose ...

Small-sized data transfers

When transferring a modest number of smaller-sized files, you can also use:

You can additionally use protocols like FTPS and tools like Rsync for this purpose ...

Transfers to/from repositories under version control

When your code and/or data are stored in repositories under version control, client software is available for accessing them via:

  • Git
  • Mercurial
  • Subversion (SVN)

See Accessing Software for information on finding and loading this software via the BRC supercluster's Environment Modules.

Transfers to/from specific systems

For bDrive (Google Drive) and Box, we recommend using rclone to transfer data to and from Savio. 

  • See here for instructions for using Box and bDrive with Savio.
  • See here for additional information about bDrive (Google Drive) provided by our Research Data Management program.

Additional tutorials for transferring files to/from Amazon Web Services (AWS) S3 and other popular data storage systems are in planning or development. If you have any interest in working on or testing one of these, or have suggestions for other data transfer tutorials, please contact us via our Getting Help email address!

Tags: hpc All Tags