/tag/storage

  • Service Status

    Project Storage Data Migration The storage hardware serving the Research Project Storage file system has been experiencing intermittent disruptions since October 2023. In response, we acquired and installed new, upgraded hardware offering faster, more reliable access and capacity. However, because an Active File Management (AFM) connection between the old and new systems was used to facilitate automatic file transfers, the new system’s performance is being negatively impacted by the old hardware and causing accessibility issues for some users.
    To mitigate these issues, Research Computing engineers are switching to an alternate backend migration process on February 26, 2024, at 9:00 a.m.

  • Rivanna

    Rivanna is the University of Virginia’s High-Performance Computing (HPC) system. As a centralized resource it has hundreds of pre-installed software packages available for computational research across many disciplines. Currently the Rivanna supercomputer has 603 nodes with over 20476 cores and 8PB of various storage.
    All UVA faculty, staff, and postdoctoral associates are eligible to use Rivanna, or students when part of faculty research.
    Facilities Statement - Are you submitting a grant proposal and need standard information about UVA research computing environments? Get it here. The sections below contain important information for new and existing Rivanna users. Please read each carefully.

  • Globus Data Transfer

    Globus is a simple, reliable, and fast way to access and move your research data between systems. Globus allows you to transfer data to and from systems such as:
    Laptops & personal workstations Rivanna HPC cluster Ivy Central Storage Lab / departmental storage Tape archives Cloud storage Off-campus resources (ACCESS, National Labs) Globus can help you share research data with colleagues and co-investigators, or to move data back and forth between a lab workstation and Rivanna or your personal computer.
    Are your data stored at a different institution? At a supercomputing facility? All you need is your institution’s login credentials.

  • Drive Mapping on Windows or Mac OSX

    Research Standard and Research Project storage can be mapped to your Windows or Mac computer as a remote drive. If you are off Grounds you must be running a VPN, such as the UVA Anywhere or the More Secure VPN from ITS. We recommend the More Secure VPN if that is available to you.
    Windows Open a File Explorer page. In the left column, right click on This PC. In the drop-down box that appears, look for the Map Network Drive option. If you do not see this option, click on Show more Options, and then click on Map network drive….

  • Mission

    Research Computing empowers UVA researchers to achieve more with cutting-edge computational resources. Our support team strives to create innovative solutions for researchers who need help solving complex optimization, parallelization, workflow, and data analysis issues. We build and maintain the University’s best computing platforms while educating the next generation of researchers on the power of advanced computing.
  • Commercial Data Sharing and Archiving Solutions

    Box® Non-sensitive cloud storage UVA Box is a cloud-based storage and collaboration service that gives eligible members of the University community the ability to access, store, and share up to 1 TB of public / internal use University files securely—anywhere, anytime, on any device. Read more DropBox®/Sookasa® Highly Sensitive Data (PHI/PII) storage If you plan on storing highly sensitive data such as PHI or PII, UVA Health System offers a secure encrypted storage for Health System affiliated researchers, students, and staff. “DropBox Sookasa” is a free cloud-based service hosted on Dropbox that can be accessed over the internet on any device.
  • Rivanna Storage

    There are a variety of options for storing large-scale research data at UVa. Public and Internal Use data storage systems can be accessed from the Rivanna high performance computing system.
    .tg {border-collapse:collapse;border-spacing:0;border-color:#ccc;} .tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#fff;} .tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:0px;overflow:hidden;word-break:normal;border-color:#ccc;color:#333;background-color:#f0f0f0;} .tg .tg-hy9w{background-color:#eceeef;border-color:inherit;vertical-align:top} .tg .tg-dc35{background-color:#f9f9f9;border-color:inherit;vertical-align:top} .tg .tg-0qmj{font-weight:bold;background-color:#eceeef;border-color:inherit;vertical-align:top} Storage Directories Name Quota Price Data Protection Accessible from Best Practices /home 50GB Free 1-week snapshots Rivanna /home is best used as a working directory when using Rivanna interactively. Slurm jobs run against /home will be slower than those run against /scratch. The /home directory is a personal storage space that is not shareable with other users.

  • Research Standard Storage

    Overview Research Computing offers several budget options for storing public and internal use research data. Information Security at UVA provides details about data sensitivity
    classifications.
    The Research Standard storage provides users with a solution for research data storage and collaboration. Members in the same group have access to a shared directory created by
    the team lead or PI. Group membership can be defined and managed through Grouper (requires VPN connection). Research Standard storage is mounted on the Rivanna HPC cluster and can also be accessed on a personal computer with an SMB
    mount, allowing for point-and-click file manipulation.
    If you are not a researcher, UVA ITS offers Value storage for long-term storage of large scale data.

  • Integrated Sensor Networks in Arctic Alaska

    Understanding the Changing Natural-Built Landscape in an Arctic Community Navigating the New Arctic (NNA) is one of The National Science Foundation’s 10 Big Ideas. NNA projects address convergence scientific challenges in the rapidly changing Arctic. Arctic research is needed to inform the economy, security and resilience of the Nation, the larger region and the globe. NNA empowers new research partnerships from local to international scales, diversifies the next generation of Arctic researchers, enhances efforts in formal and informal education, and integrates the co-production of knowledge where appropriate. This award fulfills part of that aim by addressing interactions among social systems, natural environment, and built environment in the following NNA focus areas: Arctic Residents, Data and Observation, Education, and Resilient Infrastructure.
  • Cloud Storage Solutions

    Amazon Web Services Tiered object storage Amazon S3 and Glacier offer cloud-based, affordable, unlimited capacity for storage from anywhere. Advanced features include scalability, lifecycle management, encryption, and sharing. S3 is ideal for static files that need to be retrieved from any location (PDFs, images, video, etc.). Glacier is archival storage, perfect for grant compliance that reqires data retention. How RC can help: Lower pricing - UVA has an Internet2 discount available for educational use. Contact us to create an account for you or your research project. Cost estimates - Cloud storage is not free. Consideration should be made to the size of your files and how often they will be retrieved.
  • ITS `home1` directories unavailable on Rivanna

    ITS has completed phase 1 of its network reconfiguration and ITS NAS storage volumes have been remounted on Rivanna. RC managed Research Standard and Research Project storage are also fully available. ITS home1 directories, including /nv/t* mounts, remain unavailable on Rivanna. Users will still be able to mount ITS home1 storage to their local workstations and transfer their data via Globus and the UVA Main DTN. We regret the inconvenience.
  • Pricing

    Below is a schedule of prices for Research Computing resources.
    Rivanna Allocations Type SU Limits Cost SU Lifetime Standard None Free 12 months Purchased None $0.01 Forever Instructional 100,000 Free 2 weeks after last teaching session About Allocations
    Storage Name Security Cost Research Project Standard $60 TB/year Research Standard Standard $45 TB/year ZFS Standard $30 TB/year – Ivy Central Storage High $45 TB/year Ivy NAS Storage High $60 TB/year – Storage Details Request Storage

  • Transfer Files Using Amazon S3

    Setup You will need to install and configure the awscli package in order to access objects in S3.
    Install the AWS CLI The AWS CLI is available through the pip/pip3 installer:
    If you have administrator privileges type
    pip install awscli $('#copybtn891632745').click(function(){ var $temp = $(""); $(“body”).append($temp); $temp.val($('#891632745').text()).select(); document.execCommand(“copy”); $temp.remove(); var $this = $(this); $(this).fadeOut(200); $(this).delay(2000); $(this).fadeIn(200); }); Otherwise use
    pip install –user awscli $('#copybtn685472139').click(function(){ var $temp = $(""); $(“body”).append($temp); $temp.val($('#685472139').text()).select(); document.execCommand(“copy”); $temp.remove(); var $this = $(this); $(this).fadeOut(200); $(this).delay(2000); $(this).fadeIn(200); }); The project is open source, so you can also download the source at https://github.

  • ACCORD: Jupyter Lab

    Back to Overview
    Jupyter Lab allows for interactive, notebook-based analysis of data. A good choice for pulling quick results or refining your code in numerous languages including Python, R, Julia, bash, and others.
    Learn more about Jupyter Lab

  • ACCORD: RStudio

    Back to Overview
    RStudio is the standard IDE for research using the R programming language.
    Learn more about RStudio

  • ACCORD: Theia IDE

    Back to Overview
    Theia Python is a rich IDE that allows researchers to manage their files and data, write code with an intelligent editor, and execute code within a terminal session.
    Learn more about the Theia Python IDE

  • How To Tips for Storage

    Map your Research Standard or Research Project Storage to your Desktop Use Globus from the Command Line Work with files in Amazon S3
  • Globus Command-Line Interface

    Install the Globus CLI The Globus CLI is available through the pip installer:
    If you have administrator privileges type
    pip install globus-cli Otherwise use
    pip install –user globus-cli The project is open source, so you can also download the source at https://github.com/globus/globus-cli
    If you would like to use the CLI from Rivanna, please follow these directions below.
    Authenticate using the Globus CLI Log in against your institutional authentication provider. In the case of UVA, we use NetBadge for signing in:
    globus login This will open a page in your web browser where you select your institution and proceed to login:

  • Storage FAQs

    Accidental File Deletions Why Lease Storage Leased Storage Options File Transfer with Globus Permission Denied Error with Globus Help! I deleted my files accidentally! What can I do? Your home directory contains a hidden directory .snapshot which is not visible even with ls -a. Change into that directory to find timestamped versions of your home directory. Snapshots are created once per day. For GPFS Research Project (leased) storage, the directory is /gpfs/gpfs0/project/.snapshots. For non-GPFS Research Project storage, the directory is /project/.snapshot. Neither Research Standard (leased) nor Rivanna scratch storage is backed up in any way. Why should I lease storage?
  • Frequently Asked Questions

    General Do you have a general computing question?
    Read our FAQ› Rivanna High Performance Computing Platform
    Read our FAQ › Ivy Secure Data Computing Platform
    Read our FAQ › Storage Research Data Storage & Transfer
    Read our FAQ ›

  • How To

    General General tips and tricks for computational research.
    General HowTos › Rivanna High Performance Computing platform
    Rivanna HowTos › Ivy Secure Data Computing Platform
    Ivy HowTos › Storage Research Data Storage & Transfer
    Storage HowTos ›

  • Research Data Storage

    There are a variety of options for storing large-scale research data at UVA. Public and internal use data storage systems can be accessed from the Rivanna high performance computing system. Sensitive and highly sensitive data can be stored and accessed within the Ivy secure computing environment. Information Security at UVA provides an overview of the data sensitivity classifications. UVA graduate and undergraduate students are not permitted to request storage accounts. This must be done by their faculty advisor[s].
    Information Technology Services (ITS) also provides multiple tiers of data storage for personal and non-research storage needs. .

  • Public and Moderately Sensitive Data Storage

    /home /home is a free 50GB space provided to users of the Rivanna HPC system and is visible from the Rivanna login and compute nodes. /home is the default working directory when logging on to Rivanna. Users can also access their home directory at /home/$USER, where $USER is an individual’s UVa computing ID.
    /scratch /scratch is a Lustre high performance parallel filesystem accessible via the Rivanna login and compute nodes.
    Rivanna’s scratch file system has a limit of 10TB per user. This policy is in place to guarantee the stability and performance of the scratch file system. Scratch is intended as a temporary work directory.

  • Slurm Job Manager

    var cursor = true; var speed = 280; setInterval(() = { if(cursor) { document.getElementById(‘cursor’).style.opacity = 0; cursor = false; }else { document.getElementById(‘cursor’).style.opacity = 1; cursor = true; } }, speed); // Add event listener on keydown document.addEventListener(‘keydown’, (event) = { var name = event.key; if ( name === ‘y’ ) { window.location.href = “/quiz/slurm/"; } else if ( name === ‘n’ ) { $('#slurm-modal').modal(‘hide’); } }, false); SLURM Would you like to take an interactive SLURM quiz? y/N |
    Overview Rivanna is a multi-user, managed environment. It is divided into login nodes (also called frontends), which are directly accessible by users, and compute nodes, which must be accessed through the resource manager.

  • Sensitive Data Storage - Ivy

    Overview The Ivy secure computing environment meets both HIPAA- and CUI-compliance standards and is ideal for storing highly sensitive research data. Ivy offers several storage options to fit your research computing needs.
    Ivy Central Storage Ivy Central Storage (ICS) is a highly sensitive data parking zone and central storage pool with a capacity greater than 1PB. This storage space is available for researchers with highly sensitive data and can be mounted on an Ivy virtual machine (VM). For added security, files stored on ICS are read & write only. Executable files can be moved from ICS to VM storage.
    Researchers can request space on ICS by first requesting an Ivy account using the Ivy request form.

  • User Guides

    High Performance Computing Standard and high security HPC to run your code, generally written in R, Python or shell scripts.
    Get Started › Secure Computing Secure virtual machines and interactive notebooks for processing HIPAA and other highly sensitive data.
    Get Started › Storage Need large, or extremely large storage offsite or on grounds? Can you count in GB, TB, or PB? Learn more about storage options and pricing.
    Get Started › Cloud Have an idea you’d like to test? Need an environment provisioned in short-order? We can help you build in the AWS cloud.