Moving Big Data

Moving Big Data

Research Computing works with researchers in the UVA Center for Public Health Genomics, to transfer large genomics datasets from partner institutions. Using Globus, an asynchronous data transfer utility (created at Argonne Laboratory and based on GridFTP), transfers of data larger than 40TB has been made easier and more reliable.

Such large transfers benefit from dedicated, high-speed connectivity between Internet2 member institutions like UVA, Cornell University, and Washington University in St. Louis.

In practical terms, Globus allows users to queue large files for transfer between servers, lab workstations, laptops, or HPC systems. Transfer is attempted for up to 24 hours, and you are notified upon completion or failure of the request. Globus can be used via a web browser, command-line utility, or a Python SDK.

Learn more about Globus.

UVA has Globus Data Transfer Nodes (DTNs) for both normal and highly-sensitive data. Researchers can learn more specifics about Globus and how to use it by visiting our Globus documentation on Discourse.