Skip to content

Authenticating rclone to Transfer Data Between Savio and Your UC Berkeley bDrive Account

Here are the steps for setting up a bDrive account for access via rclone from Savio.

Note

Research IT strongly recommends that rclone be used with a Special Purpose Account (SPA), and not with the bDrive storage owned by (and accessible via) your personal CalNet ID. Separating this third-party tool’s access from the login you may use to store sensitive data or intellectual property (e.g., papers and monographs in progress; FERPA-protected student information; etc.) is an effective way of safeguarding your files. Also, use of a SPA account conveniently enables desired access by current -- and future -- colleagues, successors, tech support personnel, et al.

Note

You can set up access to multiple bDrive accounts by following the instructions below and entering unique names at the name prompt below.

We recommend that you run the following steps from within our OOD Desktop App service. The reason is that in step 6 below, you'll need to authenticate to Google Drive in a browser and running the rclone config process and the browser authentication both with OOD Desktop makes it easier to copy the needed authentication text to the rclone config session. However, as discussed below, you could connect to the DTN in a separate terminal window and run the browser on your own machine (where you'll need rclone installed, ideally using the same version of rclone as is on Savio).

1. Start up a Savio OOD Desktop App service session and ssh to the DTN, where rclone is installed. (Alternatively, if you'll use a browser on your own machine in Step 6, you can ssh to Savio and then to the DTN.)

[paciorek@n0002 ~]$ ssh dtn

2. Configure rclone to access the bDrive account by setting up a new 'remote':

[paciorek@dtn ~]$ rclone config
2019/02/13 16:51:04 NOTICE: Config file "/global/home/users/paciorek/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

3. Name the remote a unique name; you can use "bDrive" if only planning to access one bDrive account, but otherwise choose a unique name for each bDrive account. Then enter the number corresponding to Google Drive (17 in this case).

name> bDrive
Type of storage to configure.
Choose a number from below, or type in your own value

[...snip...]
15 / FTP Connection
\ (ftp)
16 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
17 / Google Drive
\ (drive)
18 / Google Photos
\ (google photos)
19 / Hadoop distributed file system
\ (hdfs)
20 / Hubic
\ (hubic)
[...snip...]

Storage> 17
** See help for drive backend at: https://rclone.org/drive/ **

4. Now answer a few questions. (If speed is a concern, you may want to set up a Google Application Client Id, as suggested.)

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
Google Application Client Secret
Setting your own is recommended.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\ "drive"
2 / Read-only access to file metadata and file contents.
\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ "drive.metadata.readonly"
scope>
ID of the root folder
Leave blank normally.

Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.

Note that if this is blank, the first time rclone runs it will fill it
in with the ID of the root folder.

Enter a string value. Press Enter for the default ("").
root_folder_id>
Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Enter a string value. Press Enter for the default ("").
service_account_file>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n

5. When asked to use 'auto config', make sure to say 'No', because you need to authenticate with Google via a browser on your local machine (e.g., your laptop.).

Use auto config?
Say Y if not sure
Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n

6. At this point, you're at the authentication step. You'll see this prompt.

Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
rclone authorize "drive"
Then paste the result.
Enter a value.

In order to obtain the value to enter, do the following. You'll need to start a terminal on a machine with a browser available. If using the OOD Desktop app (as we recommend), open a new terminal window in the OOD Desktop. If using your own machine, install rclone on that machine; installation will produce an executable called `rclone` (Mac/Linux) or `rclone.exe` (Windows). Run rclone authorize "drive" in the terminal as follows. If using OOD Desktop app you will probably need to start the Firefox browser in the Desktop yourself and go to the link indicated (the link here is just example - make sure to fill in your own).
[paciorek@n0002 ~]$ rclone authorize drive
2022/07/01 15:32:56 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=ynpI5lWCT05AUHq238cAg
2022/07/01 15:32:56 NOTICE: Log in and authorize rclone for access
2022/07/01 15:32:56 NOTICE: Waiting for code...
2022/07/01 15:34:20 NOTICE: Got code
Paste the following into your remote machine --->
{access_token:ya29.A0ARrdaM-AeTSNHfutneXsHSyyyGL4S2KdIL4-XGBRj8WYezmECfze0z3oi3w9bhyseO3yzpT-XYHzmRI2rnUh2D0W9v2wgVqgm-lBRTH3QuA5rDqSEHAiGV_DYcmLm-2342sd90DFshxMrYUNnWUtBVEFTQVRBU0ZRRl91NjFWdFRBS1FXR29qdElpUnh2TElkOEhydw0163,token_type:Bearer,refresh_token:1//06r-htFlivTPGCgYIARAAGAYSNwF-L9IrLqkDje8JFdKQXpoRxKvUh0tngiz6I6kPEJRTj4qd6emy4bPIrDxhBNmIn_42fVeIYaA,expiry:2022-07-01T16:34:19.314138579-07:00}

7. Paste that string (the string below is just an example - make sure to fill in your own) in as requested:

Option config_token.
For this to work, you will need rclone available on a machine that has
a web browser available.
For more help and alternate methods see: https://rclone.org/remote_setup/
Execute the following on the machine with the web browser (same rclone
version recommended):
rclone authorize "drive"
Then paste the result.
Enter a value.
{access_token:ya29.A0ARrdaM-AeTSNHfutneXsHSyyyGL4S2KdIL4-XGBRj8WYezmECfze0z3oi3w9bhyseO3yzpT-XYHzmRI2rnUh2D0W9v2wgVqgm-lBRTH3QuA5rDqSEHAiGV_DYcmLm-2342sd90DFshxMrYUNnWUtBVEFTQVRBU0ZRRl91NjFWdFRBS1FXR29qdElpUnh2TElkOEhydw0163,token_type:Bearer,refresh_token:1//06r-htFlivTPGCgYIARAAGAYSNwF-L9IrLqkDje8JFdKQXpoRxKvUh0tngiz6I6kPEJRTj4qd6emy4bPIrDxhBNmIn_42fVeIYaA,expiry:2022-07-01T16:34:19.314138579-07:00}

8. You can now choose whether to configure this as a team drive. We'll enter 'No' for a basic setup.

Configure this as a team drive?
y) Yes
n) No
y/n> n

9. At this point, you'll see something like this, and you can accept the addition of the remote and quit out of the configuration.

[bDrive]
type = drive
token = {access_token:ya29.A0ARrdaM-AeTSNHfutneXsHSyyyGL4S2KdIL4-XGBRj8WYezmECfze0z3oi3w9bhyseO3yzpT-XYHzmRI2rnUh2D0W9v2wgVqgm-lBRTH3QuA5rDqSEHAiGV_DYcmLm-2342sd90DFshxMrYUNnWUtBVEFTQVRBU0ZRRl91NjFWdFRBS1FXR29qdElpUnh2TElkOEhydw0163,token_type:Bearer,refresh_token:1//06r-htFlivTPGCgYIARAAGAYSNwF-L9IrLqkDje8JFdKQXpoRxKvUh0tngiz6I6kPEJRTj4qd6emy4bPIrDxhBNmIn_42fVeIYaA,expiry:2022-07-01T16:34:19.314138579-07:00}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
bDrive bDrive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Now you should be all set to copy files to/from Savio and bDrive.

Alternative to using a SPA account for rclone data transfers

If you are working alone, with no colleagues and no need to make your backups available to others should you leave UC Berkeley, rclone provides an "access scope" setting that allows you to use your individual bDrive account while maintaining the safety of your other files by isolating them from the rclone application. This feature is available for rclone version 1.40 and higher.

To utilize this feature, you will need to grant a limited access scope to rclone during the configuration process, as follows:

  • During the configuration process you will be asked to choose the Scope that rclone should use when requesting access from drive
  • Choose the scope Access to files created by rclone only ... drive.file instead of Full access all files ... drive.

This choice will instruct bDrive to grant access only to files that have been added to bDrive using rclone; and rclone will not be able to "see," move, delete, or transfer files added to bDrive using the Google Drive web browser interface.

If using this configuration, you may find it convenient to create a single folder within which rclone can add files and folders. A command that writes to the folder rclone-bkp (creating it if the folder doesn't exist yet), in the account for which the rclone configuration is named my-bdrive might look something like this:

rclone copy /home/mylogin/Documents/bkp my-bdrive:rclone-bkp

To use a sub-folder somefiles in the rclone-bkp folder:

rclone copy /home/mylogin/Documents/bkp/somefiles my-bdrive:rclone-bkp/somefiles