Skip to content

Data transfer within Savio

Transferring data between directories within Savio brings up some different considerations than transferring data between Savio and other locations.

First, it’s helpful to know a bit about the filesystem structure on Savio:

  • All personal scratch directories and all condo storage project directories are on the DDN filesystem, mounted at /global/scratch.
  • All personal home directories, group directories, and software installed systemwide are on the VAST filesystem, mounted at /global/home and /global/software.

Moving data within a filesystem on Savio

If you are moving data between two locations within /global/scratch (or between two locations within /global/home) then you are moving data within a given filesystem. This can be done essentially immediately (unless unless moving a very large number of files) if you use mv to move the data rather than cp or rsync to make another copy. Using mv doesn’t actually change the location of the data on disk, it just updates the metadata concerning the path to the data. As an example, to move data from a personal scratch directory to a condo project directory, just use mv /global/scratch/users/<username>/mydir /global/scratch/projects/<projectname>/.

If instead you want to make a copy then see the next topic.

Copying data within Savio

If you are copying data, please follow the suggestions below. Copying would occur under the following circumstances:

  1. You need a second copy within the same filesystem (e.g., on /global/scratch).
  2. You need to move or copy data between filesystems (i.e., between /global/scratch and /global/home).

First, our primary recommendation for large amounts of data is to use Globus for such internal copies, choosing ucb#brc as both of the endpoints. Globus has various advantages, including the ability to copy individual files in parallel, to copy multiple files in parallel, to robustly restart interrupted transfers, and to notify you when a transfer is done. One can optionally configure Globus to sync files so that only differences are copied by clicking the “Transfer & Sync Options” tab between the two Start buttons on the File Manager page and choosing the “sync” option.

Second, one can instead use rsync. rsync doesn’t have the advantages of Globus mentioned above, but it is another good tool and has the nice feature of only copying the differences between the files in the two locations (hence the 'sync' in 'rsync').

Finally, one can use standard UNIX cp and mv commands.