/tag/containers

  • Microservice Deployments

    Kubernetes is a container orchestrator for both short-running (such as workflow/pipeline stages) jobs and long-running (such as web and database servers) services. Containerized applications running in the UVARC Kubernetes cluster are visible to UVA Research networks (and therefore from Rivanna, Skyline, etc.). Web applications can be made visible to the UVA campus or the public Internet. Kubernetes Research Computing runs microservices in a Kubernetes cluster that automates the deployment of many containers, making their management easy and scalable. This cluster will eventually consist of several dozen instances, >2000 cores and >2TB of memory allocated to running containerized services. It will also have over 300TB of cluster storage and can attach to both project and standard storage.
  • How to add packages to a container?

    Basic Steps Strictly speaking, you cannot add packages to an existing container since it is not editable. However, you can try to install missing packages locally. Using python-pip as an example: singularity exec <container.sif> python -m pip install –user <package> Replace <container.sif> with the actual filename of the container and <package> with the package name. The Python package will be installed in your home directory under .local/lib/pythonX.Y where X.Y is the Python version in the container. If the installation results in a binary, it will often be placed in .local/bin. Remember to add this to your PATH: export PATH=~/.local/bin:$PATH You should be able to use the new package/binary in the container, as your entire home directory is mounted at runtime.
  • Computing Environments at UVA

    Rivanna The primary vehicle for high-performance computing since 2014 has been the Rivanna cluster. Rivanna is a heterogenous system with a total of 575 nodes and 20476 cpu cores. It consists of 527 nodes with 20-40 cores and 128-768GB of RAM each, 11 large memory nodes with 16-48 cores and 1-1.5TB of RAM each, and 29 nodes with a total of 172 NVIDIA GPU accelerators (K80, P100, V100, A100, RTX-2080 Ti). In addition, 8 KNL nodes with 64 physical cores and 192GB of RAM are available. These nodes are partitioned for various types of workloads, but include development, parallel, HTC and instructional partitions.
  • Workshops

    UVA Research Computing provides training opportunities covering a variety of data analysis, basic programming and computational topics. All of the classes listed below are taught by experts and are freely available to UVa faculty, staff and students. Upcoming Workshops Workshops have been moved to online learning formats due to COVID-19. DATE WORKSHOP INSTRUCTOR Oct 10, 2022 Customizing Shiny Apps Christina Gancayco Oct 13, 2022 Parallel Matlab on Rivanna Ed Hall Oct 20, 2022 GPU-Enabled Applications on Rivanna Ruoshi Sun, Ed Hall, Jacalyn Huband, Christina Gancayco Oct 27, 2022
  • Container Service Request

    – Name * E-mail * User ID * Classification * - Select -FacultyStaffPostdoctoral AssociateOther Affiliation * - Select - College of Arts & Sciences School of Data Science School of Engineering and Applied Sciences School of Medicine Darden School of Business UVA Health System Other Project Summary Please describe your project and the container images you want to run. Tier of Service *     6 - 15 containers ($10/month total)   15 containers ($48/month total) Billing Tiers are selected and paid for by the PI.
  • BART Web

    BART (Binding Analysis for Regulation of Transcription) Web Working with researchers in the Zang Lab in the Center for Public Health Genomics (CPHG), RC helped launch BARTweb, an interactive web-based tool for users to analyze their Genelist or ChIP-seq datasets. BARTweb is a containerized Flask front-end (written in Python) that ingests files and submits them to a more robust Python-based genomics pipeline running on Rivanna, UVA’s high performance computing cluster (HPC). This architecture – of a public web application that uses a supercomputer to process data – is a new model for UVA, and one that eases the learning curve for researchers who may not have access to an HPC system or the expertise to run a BART pipeline in the command-line.
  • Computing Systems

    UVA Research Computing can help you find the right system for your computational workloads. From supercomputers to HIPAA secure systems to cloud-based deployments with advanced infrastructure, various systems are available to researchers. Are you submitting a grant proposal and need standard information about UVA research computing environments? Get it here. High Performance Computing - Rivanna A traditional high performance cluster with a resource manager, a large file system, modules, and MPI processing. Get Started on Rivanna Secure Computing for Highly Sensitive Data - Ivy A multi-platform, HIPAA-compliant system for secure data that includes dedicated virtual machines (Linux and Windows), JupyterLab Notebooks, and Apache Spark.
  • Cloud Solutions

    // Run your Cloud computing is ideal for running flexible, scalable applications on demand, in periodic bursts, or for fixed periods of time. UVA Research Computing works alongside researchers to design and run research applications and datasets into Amazon Web Services, the leader among public cloud vendors. This means that server, storage, and database needs do not have to be estimated or purchased beforehand – they can be scaled larger and smaller with your needs, or programmed to scale dynamically with your application. Service Oriented Architecture A key advantage of the cloud is that for many services you do not need to build or maintain the servers that support the service – you simply use it.
  • Rivanna HPC Software

    Overview Research Computing at UVA offers a variety of standard software packages for all Rivanna users. We also install requested software based on the needs of the high-performance computing (HPC) community as a whole. Software used by a single group should be installed by that group’s members, ideally on leased storage controlled by the group. Departments with a set of widely-used software packages may install them to the lsp_apps space. The Research Computing group also provides limited assistance for individual installations. For help installing research software on your PC, please contact Research Software Support at res-consult@virginia.edu. Software Modules and Containers Software on Rivanna is accessed via environment modules or containers.
  • Software Containers

    Overview Containers bundle an application, the libraries and other executables it may need, and even the data used with the application into portable, self-contained files called images. Containers simplify installation and management of software with complex dependencies and can also be used to package workflows. Singularity is a container application targeted to multi-user, high-performance computing systems. It interoperates well with Slurm and with the Lmod modules system. Singularity can be used to create and run its own containers, or it can import Docker containers. Creating Singularity Containers To create your own image from scratch, you must have root privileges on some computer running Linux (any version).
  • Docker - The Basics

    What Is Docker? “Docker is a set of platform-as-a-service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. Containers are isolated from one another and bundle their own software, libraries and configuration files; they can communicate with each other through well-defined channels. All containers are run by a single operating-system kernel and are thus more lightweight than virtual machines. The service has both free and premium tiers. The software that hosts…” Continue reading on Wikipedia Click to watch on YouTube:

    Install Docker Docker is available for Windows, Mac, and Linux. Download the appropriate Docker Edition for your platform directly from Docker.


  • LOLAweb

    The past few years have seen an explosion of interest in understanding the role of regulatory DNA. This interest has driven large-scale production of functional genomics data resources and analytical methods. One popular analysis is to test for enrichment of overlaps between a query set of genomic regions and a database of region sets. In this way, annotations from external data sources can be easily connected to new genomic data. SOM Research Computing is working with faculty in the UVA Center for Public Health Genomics to implement LOLAweb, an online tool for performing genomic locus overlap annotations and analyses. This project, written in the statistical programming language R, allows users to specify region set data in BED format for automated enrichment analysis.
  • Refgenie: A Reference Genome Resource Manager

    Reference genome assemblies are essential for high-throughput sequencing analysis projects. Typically, genome assemblies are stored on disk alongside related resources; e.g., many sequence aligners require the assembly to be indexed. The resulting indexes are broadly applicable for downstream analysis, so it makes sense to share them. However, there is no simple tool to do this. Refgenie is a reference genome assembly asset manager. Refgenie makes it easier to organize, retrieve, and share genome analysis resources. In addition to genome indexes, refgenie can manage any files related to reference genomes, including sequences and annotation files. Refgenie includes a command line interface and a server application that provides a RESTful API, so it is useful for both tool development and analysis.
  • ACCESS: Advanced Cyberinfrastructure Coordination Ecosystem: Services and Support

    The NSF’s ACCESS (Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support) program builds upon the successes of the 11-year XSEDE project, while also expanding the ecosystem with capabilities for new modes of research and further democratizing participation. ACCESS Home:  access-ci.org  access-ci.org/about Allocations Allocations: allocations.access-ci.org Documentation Support: support.access-ci.org Community Engagement ACCESS: support.access-ci.org/affinity-groups Campus Champions: https://campuschampions.cyberinfrastructure.org UVa Research Computing has two Champions, Ed Hall and Katherine Holcomb For more help, please feel free to contact RC staff to set up a consultation or visit us during office hours.
  • Docker Images on Rivanna

    Docker requires sudo privilege and therefore it is not supported on Rivanna. To use a Docker image you will need to convert it into Singularity. Convert a Docker image There are several ways to convert a Docker image: Download a remote image from Docker Hub Build from a local image cached in Docker daemon Build from a definition file (advanced) Instructions are provided in each of the following sections. Docker Hub Docker images hosted on Docker Hub can be downloaded and converted in one step via the singularity pull command: module load singularity singularity pull docker://account/image Use the exact same command as you would for docker pull.
  • XSEDE: Extreme Science and Engineering Development Environment

    XSEDE’s Mission was to substantially enhance the productivity of a growing community of scholars, researchers, and engineers through access to advanced digital services that support open research; and coordinate and add significant value to the leading cyberinfrastructure resources funded by the NSF and other agencies. — The XSEDE project ended on August 31, 2022 and was succeeded by the  ACCESS project. XSEDE Home:  www.xsede.org
  • Keras on Rivanna

    Description Keras is a high-level neural networks application programming interface (API), written in Python and capable of running on top of TensorFlow, CNTK, or Theano. On Rivanna, we provide TensorFlow containers that include the Keras API. Since version 1.12.0, TensorFlow contains its own Keras API implementation as described on the TensorFlow website. Using Keras with TensorFlow containers Like TensorFlow itself, Python code that utlizes the Keras package can be run interactively as Jupyter Notebooks, in interactive shell jobs, or non-interctively as Slurm batch jobs. Rivanna provides several nodes with graphics processing units (GPUs) that should be used when running Keras code.
  • Rivanna HPC Software

    Overview Research Computing at UVA offers a variety of standard software packages for all Rivanna users. We also install requested software based on the needs of the HPC community as a whole. Software used by a single group should be installed by that group’s members, ideally on leased storage controlled by the group. Departments with a set of widely-used software packages may install them to the lsp_apps space. The Research Computing group also provides limited assistance for individual installations. For help installing research software on your PC, please contact Research Software Support at res-consult@virginia.edu. Software Modules and Containers Software on Rivanna is provided via environment modules or as containers.
  • Microservices

    Microservice architecture is an approach to designing and running applications. Such applications are typically run within containers, made popular in the last few years by Docker. Containers are portable, efficient, and disposable, and contain code and any dependencies in a single package. Containerized microservices typically run a single process, rather than an entire stack within the same computing environment. This allows portions of your application to be easily replaced or scaled as needed. Transition to Kubernetes - Research Computing is transitioning away from DCOS for microservice orchestration in favor of Kubernetes, the open-source orchestrator originating from Google. New deployments will be launched directly in Kuberenetes and existing DCOS deployments will be migrated over the Summer and Fall of 2022.