Authenticating rclone to Transfer Data Between Savio and Your UC Berkeley Box Account
Here are the steps for setting up a Box account for access via rclone from Savio.
Research IT strongly recommends that rclone be used with a Special Purpose Account (SPA), and not with the Box storage owned by (and accessible via) your personal CalNet ID. Separating this third-party tool’s access from the login you may use to store sensitive data or intellectual property (e.g., papers and monographs in progress; FERPA-protected student information; etc.) is an effective way of safeguarding your files. Also, use of a SPA account conveniently enables desired access by current -- and future -- colleagues, successors, tech support personnel, et al.
Note that you can set up access to multiple Box accounts by following the instructions below and entering unique names at the name> prompt.
To set up a Box account for access, follow these instructions:
1. Login to your Savio account and SSH to the dtn, where rclone is installed.
bash-3.2$ ssh paciorek@hpc.brc.berkeley.edu
[paciorek@ln003 ~]$ ssh dtn
2. Configure rclone to access the Box account by setting up a new 'remote':
[paciorek@dtn ~]$ rclone config
2019/02/13 16:51:04 NOTICE: Config file "/global/home/users/paciorek/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
3. Name the remote a unique name; you can use "Box" if only planning to access one Box account, but otherwise choose a unique name for each Box account.
name> Box
Type of storage to configure.
Choose a number from below, or type in your own value
[...snip...]
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Dropbox
\ "dropbox"
[...snip...]
Storage> 6
** See help for box backend at: https://rclone.org/box/ **
4. Now answer a few questions.
Box App Client Id - leave blank normally.
client_id> Box App Client Secret - leave blank normally.
client_secret> Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> n
For this to work, you will need rclone available on a machine that has a web browser available.
Execute the following on your machine:
rclone authorize "box"
Then paste the result below:
5. At this point, in order to get that result to paste in, you will need a machine (e.g., your laptop or desktop) where you can run a web browser and install rclone. Install rclone on that machine (more details are provided here). Installation will produce an executable called rclone (Mac/Linux) or rclone.exe (Windows).
6. Use the rclone you just installed on your own machine to run:
rclone authorize "box"
7. This will open a browser window (if your browser doesn't open automatically, then go http://127.0.0.1:53682/auth in a browser).
Login to your Box account (ideally a SPA as mentioned above) by
selecting "Use Single Sign On (SSO)" and then entering your Berkeley email address.
a. At this point, Box may allow you to enter your CalNet credentials. (Remember that to use a SPA, you need to enter a "+" before your CalNet username and then after entering your CalNet credentials, enter the SPA account you want to access.) If so, enter your credentials and proceed to Step 8.
b. Alternatively, if you are already logged in to CalNet, you may end up directly at a screen with a button 'Grant access to Box' without being shown the CalNet login screen. If you are logged in to the CalNet account associated with the Box account you want to grant access to, then go to Step 8. However, if you are logged in to your personal CalNet account but you want to grant access to a Box account associated with a SPA, then open a new incognito/private browser window and go to http://127.0.0.1:53682/auth. Now start over with the instructions here in Step 7, selecting "Use Single Sign On (SSO)" and authenticating to the SPA.
8. Click on 'Grant access to Box'.
9. At this point, you'll need to go back to the window on your laptop/desktop from which you
ran 'rclone authorize "box" ' and copy the text that Box put into that window and paste it
into the "result>" prompt in the rclone authorization session running on Savio (from Step 5 above).
The text will look something like this, but with actual credentials for the 'access_token' and 'refresh_token' values.
result> {"access_token":"xxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxx","expiry":"2019-02-13T18:08:57.899022-08:00"}
After entering the text, you'll see something like this, and you can accept the addition of the remote and quit out of the configuration.
--------------------
[Box]
type = box
token = {"access_token":"xxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxx","expiry":"2019-04-08T17:33:55.648917-07:00"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:
Name Type
==== ====
Box box
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q
Now you should be all set to copy files to/from Savio and Box.