SoS as a script organization tool

Suppose data analysis can be or has been performed with a number of scripts, the scripts can be organized into one SoS script and executed by SoS. The simplest SoS scripts are merely verbatim copies of existing scripts, each with an extra word indicating the language of the script (Fig 1A). The scripts form a single-step workflow that can be executed in the order in which they are included. Optionally, “headers” can be added to code sections to separate them into steps of one or more workflows. Numerically ordered sections in SoS scripts can be executed sequentially as steps of a single workflow (Fig 1B), whereas sections with different names can be executed separately as individual workflows (Fig 1C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Incremental SoS scripting.

(A) An SoS script integrating a Python script that converts a Microsoft Excel file to a CSV file, and an R script that plots data from the CSV file. (B) An SoS script with headers for executing two scripts as separate steps of a workflow. The section headers are numbered with optional descriptions. (C) An SoS script with headers for executing two scripts as separate workflows. (D) An SoS script that uses a global parameter and string interpolation to unify filenames. (E) An SoS script that accepts a parameter from the command line and defines input and output targets for each step. (F) An SoS script with input and output targets derived from a global parameter. SoS provides a list of format options to aid in the formatting of filenames (e.g., ‘:n’ in this example removes extensions from filenames).

https://doi.org/10.1371/journal.pcbi.1006843.g001

If a workflow must be rerun with different options or applied to different sets of data, the scripts can be generalized to use templates, which are expanded with workflow variables using Python-formatted string literals (Fig 1D–1F). Workflow variables can be defined as parameters that accept values from the command line (Fig 1E and 1F). With the input and output of steps defined at the step level, input and output files can be accessed in the scripts as the variables “_input” and “_output” (Fig 1E and 1F). The use of SoS variables to share and pass on information, as well as the use of SoS parameters as universal lightweight argument parsers for scripts in any language, represent an easy method of consolidating scripts in different languages, while abstracting common information at the workflow level. To facilitate sharing of filenames across scripting languages, SoS implements a series of format options to render filenames in different formats (e.g., file paths, basenames, extensions, quotes for special characters) (Fig 1F).

With the option of adding comments before sections and parameter definitions, SoS enables researchers to consolidate analytic steps scattered in multiple scripts into a single, well-documented SoS script. The script can be executed using a command console. It has a proper help message (comments in the script accessed by typing command option “-h”) that summarizes available subcommands (workflow names) and parameters; it can be used to reproduce the entire data analysis or selected workflows, or it can be applied to new batches of data with different parameter settings. Even without any advanced workflow features, the ability to organize multiple scripts into a single, executable SoS script simplifies the execution of multistep, multilanguage data analysis, improving documentation, version control, and project sharing.

Basic syntaxes of SoS workflows

With increasing data size and/or analysis complexity, “pipelinizing” a workflow for more efficient and robust executions becomes more relevant. As with many pipeline tools, rigorous workflows in SoS are characterized by steps connected with static or dynamic input and output or by target dependencies in which one step provides some targets that another step depends on. All words above in bold font are SoS keywords or options that are necessary to define complex workflows, e.g., Fig 1E and 1F.

More formally, SoS extends from Python 3.6 and allows for the use of arbitrary Python statements and modules inside an SoS workflow. An SoS script is just a Python script separated into multiple sections and interpreted by SoS. The script blocks with headers shown in Fig 1 are Python function calls written in a special script format so that the first parameter (the multiline script) can be easily specified, with or without string interpolation (option expand). Indentation of included scripts is optional, but recommended for clarity. An SoS section has a header and a body consisting of SoS directives to specify step input, output, dependent targets (if applicable), and arbitrary Python statements. The section header specifies names of and options for SoS steps and, together with the input, output, and dependent targets, determines how a workflow should be created from the sections.

The script format of function calls, section header, parameter, input, output, depends, and task (to be discussed later) statements constitute all the syntaxes that SoS adds to Python 3.6+. These statements accept options in Python syntax and control how SoS workflows and steps are executed. For example, an SoS step is usually executed once with variable _input representing all input files of the step (Fig 1). However, with input option group_by = 1, SoS divides the input files into groups of size 1 and executes the step statements (called substeps) multiple times with _input presenting each of these input groups. SoS provides multiple ways to specify input sources (e.g. from outputs of upstream steps) and to group input files, and by default executes substeps in parallel.

Multistyle workflow system

An SoS workflow is defined as a collection of SoS steps that are connected to form a Directed Acyclic Graph (DAG). Distinct from other pipeline tools, SoS has a multistyle design to provide more than one way to construct a workflow.

The most intuitive SoS workflow style is the process-oriented style, as used with some other pipeline tools [14, 22]. In its simplest form, this style sequentially executes a series of numerically ordered steps (Fig 1, with two single-step workflows defined in Fig 1C). The step numbers do not have to be consecutive, allowing steps to be added later, if needed. Although such a workflow, by default, has a DAG with a single linear execution path, users can break the default dependencies and introduce new dependencies (edges of DAGs) by defining input, output, and dependent targets in each step. These dependent targets can be files, other SoS steps (function sos_step), outputs from upstream steps (functions output_from and named_output), software libraries, or available system resources. SoS creates a DAG from the steps and their dependencies and execute steps with met dependencies in parallel, regardless of the logical orders in which steps are specified. This process-oriented style is most naturally applied to pipelines with dynamic or multiple known outcomes. Furthermore, it is logically intuitive and thus friendly to novice users.

Another workflow style that SoS supports is the outcome-oriented or Makefile style, which relies on implicit wild-card idioms introduced with the Make utility, and adopted by some pipeline tools [15, 23]. Harnessing the power of wild-card pattern matching, the outcome-oriented style implicitly determines dependencies, and automatically builds and executes DAGs. Triggered by the specification of one or more output files through option -t of the sos run command, outcome-oriented style specification enables the flexibility of generating specific outputs by executing only the required steps and there is not necessarily a complete workflow with a final outcome. By design, it fully parallelizes the workflow engine to schedule and execute jobs. However, the logic in the outcome-oriented style is backward, which is counter-intuitive to some bioinformaticists; the conventional pattern-matched dependency specification can be tricky for steps with no obvious or dynamic outcomes. Therefore, this style is often more challenging to use and extend than process-oriented style [14].

The multistyle design of SoS alleviates the dilemma of choosing between process- and outcome-oriented styles. With SoS, researchers have the freedom to build intuitive pipelines in the process-oriented style and, as the workflow grows, convert it to the outcome-oriented style with minimal syntax changes, even though the underlying logics of the two styles are very different. In fact, many SoS users may prefer to work with the third SoS coding style: the mixed style. In this style the trunk of the workflow can be process-oriented, with dependencies in both process-oriented (depends on other steps by step names or named or unnamed outputs) and outcome-oriented (depends on pattern-matched outputs) styles. Alternatively, the trunk of the workflow can be outcome-oriented, with dependencies generated by subworkflows consisting of multiple process-oriented steps (a step depends on another workflow). The proportions of process-oriented versus outcome-oriented steps depend on the needs of specific problems. When properly applied, the mixed style usually resolves logic and syntax dilemmas encountered with the other two styles.

Fig 2 illustrates applications of the three SoS workflow styles to the same real-world problem, a simulation study elaborated on in the Results section. In this three-step numerical comparison (apart from the “default” section, which serves as a default entry point if multiple workflows are defined in the script), step 2 depends on step 1, and step 3 depends on both step 1 and step 2. As shown in Fig 2A, SoS sequentially executes one step after another, with each step analyzing all data sets in parallel. In Fig 2B, the logic is backward, as indicated by wild-card patterns, and the steps are triggered separately for each data set. In Fig 2C, the first step is “auxiliary” in the sense that it provides data to other steps only as needed; thus, users can focus on developing and expanding the core part of the simulation study (i.e., when the regression methods are executed and evaluated).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Multistyle SoS workflows.

Shown are abstracted versions of implementations of high-dimensional regression workflows in SoS demonstrating the (A) process-oriented, (B) outcome-oriented, and (C) mixed workflow styles. These workflow style definitions are inspired by Leipzig [37], who classified workflow systems as either “explicit convention” (process-oriented) or “implicit convention” (outcome-oriented) frameworks. The workflow logics are primarily reflected by the SoS section names and step statements “input,” “output,” and “depends.” Full versions of these workflows are available online [28].

https://doi.org/10.1371/journal.pcbi.1006843.g002

In addition to the workflows with DAGs built before execution, as described above, SoS allows for construction and execution of nested workflows within another workflow step (function sos_run in Fig 2). Combined with the ability to execute any Python statement in an SoS step, and thus execute nested workflows conditionally or repeatedly for each substep applied to subsets of input files or with different parameters (Fig 2C), SoS facilitates the creation of complex, dynamic workflows in styles suitable for the complexity of a project. Another improvement of SoS over conventional outcome-oriented workflow styles is that it extends dependency targets beyond files to prerequisites for execution of steps, such as executable commands, installed libraries, and, particularly, another SoS step or workflow. For example, dependencies can be defined by step names, in addition to step output (see the Results section for an example). From an engineering point of view, this feature addresses the known issue of file-based DAGs being impractical for large applications [14]. In practice, it further improves pipeline readability and provides support for dynamic file targets.

SoS builds a DAG from explicit and implicit step dependencies to either execute a specified workflow (process-oriented) or obtain specified targets (outcome-oriented). The DAG can be specified and resolved fully before parallel execution, or with unknown dependencies that can only be verified, created, and resolved during execution. The DAG also can be expanded dynamically with the execution of nested workflows (Fig 2C). SoS uses runtime signatures to track workflow execution (via the content of steps, environments, input, output, and dependent targets) so that successfully completed steps are ignored during the re-execution of the workflow. It also protects output files using process locks during the execution of workflows, and removes partial outputs from failed steps so that output files will only be written by one SoS instance and are saved only after successful completion of the steps. Unlike time stamps [15], step signatures in SoS are generated using checksums and are workflow-independent. Therefore, completed steps can be ignored by revised or completely different workflows, as long as the signatures stay the same. Finally, SoS allows for replacement of large intermediate files with their signatures via an operation called zap. In SoS, a “zapped” step is considered complete, unless the actual intermediate files are required in later steps in the workflow.

Cross-platform workflow execution and job management

As research projects advance to stages involving large-scale data analysis consisting of numerous computationally intensive tasks, SoS remains particularly relevant owing to its unique cross-platform task model (Fig 3). Using a YAML-based configuration system that defines properties of all remote computing environments (hosts), SoS can submit tasks in the same workflow to one or multiple isolated hosts, such as high-performance computing clusters running various task queue systems (PBS/MOAB/LFS/Slurm), standalone workstations owned in a laboratory, and virtual machine instances hosted by cloud services. When a task is executed by different remote hosts, SoS translates paths of the task into paths on remote hosts, and optionally synchronizes input as output files as tasks are executed (Fig 3). With proper configuration of all available hosts, which can be done in advance by a system administrator for all users, running cross-platform computations involves simply sending tasks to hosts with appropriate hardware and software resources. SoS provides a series of command utilities to monitor and manage tasks. For example, sos status -q server lists the status of all tasks running on the specified server, and sos kill task_id terminates the execution of a task. These utilities unify job management on different platforms using the same interface on a local machine.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. SoS task model.

This diagram illustrates SoS remote task execution managed on a local machine whose file system is not shared with the remote host. 1) SoS translates the local task to a remote task via file-path mapping. 2) At the same time, SoS transfers files necessary for the computation to the remote host. 3) SoS then uses preconfigured templates to send commands to the remote job-management system to submit jobs, if applicable, or to directly execute jobs on the remote host. 4) Output files are generated at the end of computation on the remote host. 5) Results are automatically transferred back to the local host, thus completing the computational task on the local machine without direct interaction with remote hosts. Of note, different tasks can be sent to multiple remote hosts simultaneously. The local machine only provides the computational resources necessary to manage and monitor these tasks.

https://doi.org/10.1371/journal.pcbi.1006843.g003

In cases when data are large and reside on remote hosts, SoS allows the execution of tasks or complete workflows on the remote host without file synchronization. For example, the input of a step can be marked as remote, so that SoS can process files that reside on a remote host, while still copying results to the local host for downstream analysis. Similarly, output files marked as remote will stay on the remote host. When both input and output files are on the remote host, it can be easier to write workflows with respect to remote paths and use option -r host to send the entire workflow to the remote host for execution. This is equivalent to, but much more convenient than copying the workflow to the remote host, logging in to that host, and executing the workflow there. The ability to analyze data remotely using local workflows simplifies the management of projects, especially when the project consists of both local and remote data analysis steps and are recorded in the same notebook.

Container support

Whereas SoS tasks can be executed locally or distributed to multiple remote hosts, the host environment must be properly configured with required software applications. SoS addresses this problem with built-in support for docker and singularity container engines, which isolate applications and their supporting libraries and tools such that the entire tool chain and related resources are encapsulated as a bundle and can be readily executed on different platforms. By simply specifying the image and engine to use as a parameter for the script to be executed, SoS downloads the specified image, mounts appropriate directories, and executes the script inside the container. By substituting local commands and libraries with docker- or singularity-based tools [24], an SoS workflow becomes less dependent on the local runtime environment, and therefore enhances reproducibility.

Multistyle workflow demonstration

We previously demonstrated the multistyle SoS workflow design with a published computational experiment of high-dimensional regression [27]. The experiment simulates labeled data sets, builds regression models, and evaluates prediction errors for Lasso and ridge regression, as outlined in Fig 2 and detailed online with links to full scripts and commands [28].

Fig 2A illustrates the process-oriented style. The default order of execution is set by the numeric order of step indices (“_1”, “_2”, “_3”). The first section, as its name “lasso_1, ridge_1” suggests, is shared step of both Lasso and ridge regression pipelines. The core algorithms “lasso_2” (and “ridge_2,” [28]), coded in different sections, use output of the first section in groups of two for training and testing data, and produce two files for parameter estimation and prediction. Outputs from first two steps both flow into “lasso_3, ridge_3,” another shared step, for performance evaluation. Dependency on both its upstream steps is explicitly configured by named inputs and outputs (function output_from), overriding the default dependency on its immediate upstream step when evaluated by the ordering of step indices. Of note, simultaneous execution of “lasso” and “ridge” have to be explicitly specified (by using one sos_run call) because dependency between the two subworkflows is ambiguous otherwise. Fig 2B illustrates the outcome-oriented style. Expected results are specified at the beginning of the script. The “default” step requires these results, which, through wild-card pattern matching, leads to the “evaluation” step, which depends on Lasso and ridge regression [28], and finally leads to execution of the “simulation” step to generate the required data set. Unlike that for process-oriented workflow style, the backward logic herein fully specifies dependencies implicitly, and is by default parallelized to execute both regression methods.

Fig 2C illustrates the mixed workflow style. It balances clarity of logic and simplicity of implementation of the computational experiment. Given the “simulation” step, or any step capable of providing training and testing data sets, this workflow is simplified to “model fitting” and “evaluation” of Lasso and ridge regression [28]. These steps are implemented in a process-oriented style, whose data dependency, similar to the outcome-oriented style, is specified by the wild-card pattern matching the “simulate” step. This style is likely more suited for this study than the other two styles, because it encourages researchers to separate data and methods in a way that the core analysis can be applied to a number of data inputs of interest.

Additionally, we demonstrate online an example “Outcome Oriented Step Targets,” a revision of Fig 2B that uses step dependencies; and an example “Mixed Style Data Flow”, a revision of Fig 2C that uses named outputs dependencies [28]. Although the execution logic remains unchanged, these alternative implementations are possibly more intuitive compared to using wild-card idiom for the problem. We also provide online a modular version of this application, which keeps regression methods implementation in separate files and execute them from both process- and outcome-oriented workflows.

Migrating to SoS

To demonstrate migration of existing projects to SoS, we reimplemented three published bioinformatics workflows. The first workflow combines interactive analysis of differential expression, originally published as a Bioconductor tutorial but without the RNA-seq alignment workflow that we included here. The second workflow adapts a normalization and expression residual analysis pipeline for the Genotype-Tissue Expression project [29], originally written in the WDL workflow description language at the Broad institute [30]. The third workflow implements a published samtools whole genome sequencing variant calling pipeline using container tools. We demonstrate with these examples that migration to SoS requires minimal extra coding and that the codes are better organized and documented, once they are ported to SoS.

RNA-seq alignment and differential expression analysis

The Bioconductor software project provides R libraries capable of handling aligned sequence data when performing differential expression analysis, as shown in the Bioconductor tutorial. However, external software packages and data must be used to perform the alignment to generate BAM files from FASTQ files [31, 32]. Instead of using separate fragmented scripts, our SoS implementation of the alignment pipeline is coded inside the interactive analysis notebook for differential expression analysis, which downloads all required data in SRA format from the Gene Expression Omnibus, decrypts them in FASTQ format, performs read alignment, and outputs BAM files. This alignment pipeline uses dockerized tools to guarantee reproducibility. It further benefits from the use of the Jupyter IDE to embed multipage Portable Document Format figures in interactive analysis (which precludes rendering of low-quality bitmap images for tutorial presentation in addition to high-quality vector graphics for publication), and to automatically provide an HTML-based report focusing on narratives and analysis results.

RNA-seq normalization and expression residual analysis

This is an example of a data-preprocessing pipeline for the Genotype-Tissue Expression project, developed by the project’s analysis group [33]. Several pipelines available from this group generate gene expression count data and perform various types of preprocessing of expression and genotype data, followed by expression quantitative trait loci association mapping. Because raw RNA-seq data and genotypes are not publicly available, we focus on migrating the preprocessing pipeline to demonstrate cross-platform analysis of large data sets [34]. The original pipeline is written in WDL, which executes Python, R, and shell scripts, accompanied by separate text file documentation. We port this pipeline to SoS with few changes to the workhorse scripts, creating a single file containing all of the codes, documentation, and configurations for execution on remote machines with the following computational requirements: the normalization step requires more than 60 GB of memory, and the expression residual analysis [35] requires submitting 53 CPU-intensive batch jobs. In the SoS implementation, we use dedicated Markdown sections to emphasize computational challenges. Furthermore, with two lines of configuration (as options for the task statement), memory- and CPU-intensive jobs are sent to separate remote computers, with output automatically synchronized. The workflow is written in Jupyter IDE, which can be executed interactively or by using a command console, with an HTML report exported for circulation.

Whole-genome sequencing mapping and variant calling pipeline

We migrated the samtools variant calling tutorial [36] to SoS workflow. All workflow steps are executed by containers we have prepared and made publicly available. Additionally, this notebook contains information on software setup instructions, steps to prepare reference genome and variants data, and examples to review intermediate output files and visualize diagnostic plots, demonstrating seamless integration of workflow steps and possible interactive exploration of step output under a unified framework.