Bioinformatics resources
Workflow
Section titled “Workflow”A Workflow is one or more versions of a Nextflow file (.nf), a WDL file (.wdl), or a
set of files in a .zip archive.
A workflow and all of its versions can be used in any environment — any AWS account, any GCP project. You can develop a new pipeline in the Bioinfo Dev account and then process real samples with it in Bioinfo Prod.
The version can be any arbitrary string, but semantic versioning (v1.2.3) is recommended.
When a new version is registered it becomes the latest. If that version is deleted, the
previous one is promoted back to latest.
Submission
Section titled “Submission”A Submission is one execution of a workflow version. It consists of:
- the workflow and version;
- an input parameters file (JSON or YAML);
- a target engine;
- a target queue, and optionally a fallback queue.
The engine and queues must all belong to the same environment.
From GENI you can follow and manage submissions across every environment. A failed submission can be run again (retry), and a running one can be cancelled. On cancellation GENI stops the Nextflow process and every Batch job associated with that submission.
A Task is a by-product of a submission. Whatever the Batch service reports about a job — runtime, status, exit code — GENI records as a task.
Because a task can be retried on a fallback queue, several Batch jobs can belong to the same logical task. Cost and metrics are aggregated across those attempts.
Instance
Section titled “Instance”Instances are the EC2 machines or GCP VMs that Batch manages to run jobs. One instance can run many jobs at the same time. GENI records them because they are what the processing cost is actually computed from.
Tags are key/value labels. A submission can carry several tags, as long as each key
appears only once. Many submissions can share the same key and value — for example
client=acme, project=dnabr.
Tags must be registered by an administrator before they can be used. They exist to filter submissions and to aggregate cost.
Tags can also be attached to a workflow, in which case every submission of every version of that workflow inherits them. See Tagging.