This document describes how to use Globus to transfer data between your computer (or any other computer on the Internet that is acting as a Globus "endpoint" or "collection" and to which you have access) and the Berkeley Savio and Vector clusters. An "endpoint" is simply a location (e.g., laptop, desktop, server, cluster, storage system) that you want to transfer files to or from. A Globus "collection" is a discoverable access point that allows data to be transferred through GridFTP or HTTPS, and contains policies that govern sharing data with other users.
Some locations that you can use to transfer data to/from include your own personal computer, bDrive/Google Drive, Wasabi, Cloudian, AWS, the Secure Research Data and Compute (SRDC) platform at UCB, and some campus departmental facilities such as the Econometrics Laboratory and the Statistical Computing Facility.
Globus is ideal for transferring large files and/or large numbers of files at high speed. It allows you to start an unattended transfer and have confidence that it will be completed reliably; even sending you an email notification when it's done.
For information on how to use it, see Globus's Step-by-Step Getting Started Guide.
Using Globus with Savio¶
This video walks through the process of using Globus with Savio:
A quick summary for users:
- Visit the Globus website.
- Click the Log In button at upper right and log in using your institutional login (e.g. UC Berkeley or LBNL), or Globus ID. -If this your first time, you can create a free Globus ID here.
- You should land on the File Manager page.
- Turn on two-panel mode by toggling the switch near the top right.
- Use the File Manager page to transfer files:
- Click in the Collection field at left to search for or select an endpoint.
- Search for
ucb#brcin the Collection field. (This is the Globus endpoint for the BRC supercluster, which connects via the supercluster's Data Transfer Node, dtn.brc.berkeley.edu.)
- Select the
ucb#brcentry when it is displayed below. Doing so will return you to the original Transfer Files page.
- If prompted to authenticate, click the "Continue" button and follow the on-screen instructions. (In brief, you'll be taken to an LBNL page where you can enter your BRC cluster username, along with your token PIN followed by a one-time password that you will generate via the Google Authenticator application. See Logging In for details on generating that password.)
- Enter the path for the directory on the BRC cluster to/from which you'd like to copy files, into the Path field at left. (E.g.
/global/home/users/myusernamefor your home directory, where
myusernameis your BRC cluster username. For more details on storage locations, see the relevant User Guide page for the cluster on which you are working.)
- Click in the Collection field at right and repeat the process for your second endpoint. This second endpoint would often be your own computer, which you would need to set up as an endpoint as described below. (Globus also provides test endpoints,
go#ep2, each of which provides sample files that you can practice transferring to the cluster.)
- In the list of files in either of the two endpoints you have chosen (i.e., in either the left or right panel), select one or more files or folders.
- Click the appropriate "Start" button at the bottom of each endpoint to transfer these files to the other endpoint.
Transferring files between Savio and your personal computer¶
To transfer files between your computer and the BRC supercluster, you'll need to set up Globus Connect Personal. Here's a quick overview of how to install and use it:
- Log into the Globus website. (If you're not already logged in, that is.)
- Visit the Collections page.
- Click on "Get Globus Connect Personal".
- Follow the onscreen instructions on how to set up and install this software. You can also find detailed, official instructions for installing Globus Connect Personal for Mac OS X, Microsoft Windows, and Linux via links on this Globus Support "How To" page.
- Once you've installed Globus Connect Personal, and that software is actively running on your computer, go back to the "File Manager" page and select your new endpoint as above.
Transferring files between Savio and bDrive/Google Drive¶
To transfer files between your computer and the bDrive/Google Drive, follow these steps:
- In the File Manager, click in the Collection field and search for "UCB Google Drive Collections".
- When prompted to authenticate, from the Federation/Identity page, look for "University of California, Berkeley", select it, and use your CalNet credentials to log in.
- Now just navigate to your bDrive/Google Drive directory/folder of interest and transfer files in similar fashion as described in the instructions above regarding transferring between Savio and a personal computer.
Transferring files between Savio and other locations¶
Instructions for transferring to other locations, such as Wasabi and AWS, are planned. Please contact us if you'd like help in the meantime. Unfortunately at this time it is not possible to use Globus with your Berkeley Box account.