Cloud resources

These are the resources GENI Workflows provisions inside your AWS accounts and GCP projects so it can run pipelines and store data and container images.

Environment

An Environment is an AWS account plus a region (us-east-1, sa-east-1), or a GCP project plus a region (us-east4, southamerica-east1).

An organisation administrator can register as many environments as needed, each representing one account or project and one region. For example, one AWS account for Bioinfo Dev and another for Bioinfo Prod; or a single GCP project used for research and development.

Having an environment configured is what allows GENI to create the rest: engines, queues, storage, registries and plugins.

Engine

An Engine runs Nextflow or Cromwell whenever a user submits a workflow.

Each engine carries a list of S3/GCS buckets it may access, and with what permission — read only, write only, or read and write. An administrator can create as many engines as needed: for example, one for whole-genome samples and another for validation runs, each with its own list of permitted buckets.

The Nextflow process started by the engine delegates the actual bioinformatics work to a processing queue.

Queue

A Queue is where the workflow’s tasks execute, on AWS Batch or GCP Cloud Batch. Both services do the same job — run containerised work on on-demand compute instances — and GENI abstracts their differences behind one interface.

Each task defined in Nextflow or WDL becomes one Batch job. The job runs in a Docker container inside a virtual machine. The Batch service manages the instances and containers.

You select the queue at submission time, and there is no additional charge for having queues. An administrator can therefore keep a spot queue for everyday runs and an on-demand queue for urgent samples, both with the same instance family and initial storage.

AWS queues

For AWS environments an administrator can define:

• the EC2 instance types and families (m5.xlarge, r4.2xlarge, optimal);
• the initial EBS volume size — it grows on its own;
• whether the queue uses on-demand or spot instances.

Spot queues cost less than on-demand, but AWS can reclaim the instances, terminating every container running on them. Queues are configured to retry any job that failed for spot reasons up to 5 times.

Tip

For the large majority of bioinformatics tasks, spot queues give the better price/performance. Prefer on-demand for tasks that need very large machines, take more than an hour, or are on a tight deadline.

GCP queues

For GCP environments you cannot define the list of instance types to use, nor the initial VM volume size. Instead, each task declares its CPU, memory and disk requirements.

The CPU and memory requirements can be overridden with an instance type. Doing so makes GCP use that instance type rather than building a custom instance, which costs about 5% more.

Storage

Storage means Amazon S3 and Cloud Storage buckets. Workflow input parameters point at files in these buckets.

GENI can create and manage buckets for you, but you can equally use buckets that already exist in your account, buckets in other accounts, or public buckets. For those, the administrator configures the engine to allow access, and if needed adjusts the bucket policy on the other account’s side.

Registry

A Registry is an Amazon ECR registry. By default every engine configured in an environment has access to the registries in the same AWS account, plus any registries explicitly granted to that account.

Plugins

Plugins are optional extras enabled per environment. Two exist today, both AWS-only:

Plugin	What it provisions
Sentieon	A Sentieon license server — see Sentieon
Topic	An AWS SNS topic that publishes submission status changes