Managing workflows

Once you are signed in, the first step is to register a workflow and its first version, or a new version of an existing workflow.

List workflows

geni workflow list

See the versions of one workflow:

geni workflow versions <workflow-name>

Every listing command accepts --format, which takes table (the default), csv or json.

Register a workflow or version

geni workflow create \
  --workflow-name <workflow-name> \
  --workflow-version <workflow-version> \
  --file <path to the Nextflow, WDL or ZIP file>

Recommendations

Naming

Use a workflow name without spaces, so it is easy to pass around. The GitHub repository name is a good choice.

Use semantic versioning for the version (v1.2.3). Because every workflow and version is available to every environment, a suffix is a useful way to flag work in progress — v1.2.3-dev.

The Nextflow (.nf), WDL (.wdl) or ZIP file must be on your computer. The CLI uploads it to GENI, which keeps it for future submissions. Once registered, you no longer need the local file.

ZIP layout

Nextflow. The archive must contain a main.nf at its root. Every dependency must be inside the archive. Configuration files (nextflow.config) are allowed. Python scripts belong in a bin/ folder at the root — Nextflow then copies them, as executables, into the container of every task in the workflow.

WDL / Cromwell. The archive must contain a main.wdl at its root, with every dependency inside it.

zip -r <workflow-files>.zip main.nf nextflow.config modules/

Time limits

Use --cancel-time to set the maximum duration of a submission of this workflow version. After that period GENI cancels the submission automatically.

geni workflow create \
  --workflow-name wgs-germline \
  --workflow-version v2.1.0 \
  --file wgs-germline.zip \
  --cancel-time 00:12:00

The format is DD:HH:MM. There is no limit by default.

Caution

Configure --cancel-time, especially for workflows under development. Without it, a pipeline that hangs keeps burning compute until someone notices.

A submission can override this with its own --cancel-time.

Tags

Tags attached to a workflow are inherited by every submission of every version. See Tagging.

geni workflow create \
  --workflow-name wgs-germline --workflow-version v2.1.0 \
  --file wgs-germline.zip --tag client=acme --tag project=dnabr

Export a workflow

geni workflow export <workflow-name> \
  --workflow-version <workflow-version> \
  --output-folder <path>

Delete

Delete one version:

geni workflow delete \
  --workflow-name <workflow-name> \
  --workflow-version <workflow-version>

Delete a workflow and all of its versions:

geni workflow delete --workflow-name <workflow-name>

In both cases the command asks for confirmation before deleting.

Next

Managing submissions — running the workflow you just registered.