Setup You will need to install and configure the awscli package in order to access objects in S3.
Install the AWS CLI The AWS CLI is available through the pip installer:
If you have administrator privileges type
pip install awscli Otherwise use
pip install –user awscli The project is open source, so you can also download the source at https://github.com/aws/aws-cli
Rivanna Users have two options:
Load the awscli module:
module load awscli If you need a different version, install it in your user directory:
pip install –user awscli== Authenticate the CLI to Amazon Web Services Once the aws package is installed, you will need to configure it:
// Run your Cloud computing is ideal for running flexible, scalable applications on demand, in periodic bursts, or for fixed periods of time. UVA Research Computing works alongside researchers to design and run research applications and datasets into Amazon Web Services, the leader among public cloud vendors. This means that server, storage, and database needs do not have to be estimated or purchased beforehand – they can be scaled larger and smaller with your needs, or programmed to scale dynamically with your application.
Service Oriented Architecture A key advantage of the cloud is that for many services you do not need to build or maintain the servers that support the service – you simply use it.
In addition to providing free, in-person workshop training, UVA Research Computing staff teach for-credit courses. Below is a selection of courses that members of our group have taught, co-taught or provided guest lectures:
BIMS 8382: Introduction to Biomedical Data Science Spring 2017, Spring 2018
This course introduces methods, tools, and software for reproducibly managing, manipulating, analyzing, and visualizing large-scale biomedical data. Specifically, the course introduces the R statistical computing environment and packages for manipulating and visualizing high-dimensional data, covers strategies for reproducible research, and culminates with analysis of data from a real RNA-seq experiment using R and Bioconductor packages.
CS 6501: Distributed & Cloud Computing Spring 2017, Spring 2018
Rivanna The primary vehicle for high-performance computing since 2014 has been the Rivanna cluster. Rivanna is a heterogenous system consisting of approximately 186 x 20-core nodes with 128GB of RAM each, 25 x 28-core nodes with 256 GB of RAM, and 152 x 40-core nodes with 384GB each. Five “big memory” nodes offer 1TB of RAM with 16 cores each. All nodes are supported by a high-performance FDR Infiniband network using Mellanox hardware and some have EDR Infiniband. A number of nodes supporting specialty hardware are included; there are 8 nodes with NVIDIA K80 GPGPUs, 4 nodes with NVIDIA V100 GPGPUs, 1 node with NVIDIA V100, and 2 nodes with NVIDIA RTX2080TI boards.
UVA Research Computing can help you find the right system for your computational workloads. From supercomputers to HIPAA secure systems to cloud-based deployments with advanced infrastructure, various systems are available to researchers.
Are you submitting a grant proposal and need standard information about UVA research computing environments? Get it here. High Performance Computing - Rivanna A traditional high performance cluster with job scheduler, large file system, modules, and MPI processing. Get Started on Rivanna Secure Computing for Highly Sensitive Data - Ivy A multi-platform, HIPAA-compliant system for secure data that includes dedicated virtual machines (Linux and Windows), JupyterLab Notebooks, and Apache Spark.
Amazon Web Services Tiered object storage Amazon S3 and Glacier offer cloud-based, affordable, unlimited capacity for storage from anywhere. Advanced features include scalability, lifecycle management, encryption, and sharing. S3 is ideal for static files that need to be retrieved from any location (PDFs, images, video, etc.). Glacier is archival storage, perfect for grant compliance that reqires data retention. How RC can help: Lower pricing - UVA has an Internet2 discount available for educational use. Contact us to create an account for you or your research project. Cost estimates - Cloud storage is not free. Consideration should be made to the size of your files and how often they will be retrieved.