Scratch data migration

rsync

For user directories sized in the 10s of GBs and/or up to tens of thousand files, rsync is available on Savio for syncing data between /global/scratch-old and the new /global/scratch.

Examples

Sync all data, preserving permissions and links (please note the trailing slashes):

rsync -axHP /global/scratch-old/$USER/ /global/scratch/users/$USER/

Sync all data with exceptions listed in a file (excludes.txt):

# Example excludes.txt (Paths are relative to the transfer!)
exclude_dir1
excluded_file.txt
directory1/excluded_file_pattern*

# Command with exclude file:
rsync -axHP --exclude-from '/path/to/excludes.txt' /global/scratch-old/$USER/ /global/scratch/users/$USER/

More examples for excluding can be found at the following site: https://phoenixnap.com/kb/rsync-exclude-files-and-directories

Globus

Globus is a good option for users that prefer to use a GUI. Details of how to set up Globus can be found here. Please make sure to choose the endpoint of ucb#brc on both ends: /global/scratch-old/$USER/ and /global/scratch/users/$USER/.

Parallel Data Mover

rsync can become inefficient for users with directories of significant capacity (hundreds of GB) or hundreds of thousands of files or more. If your directories meet those criteria, please feel free to fill out the following form to take advantage of our parallel data mover infrastructure. Berkeley Research Computing will handle the migration accordingly: Managed Data Migration Request