systemPipeRdata: Workflow templates and sample data (original) (raw)

Introduction

systemPipeRdataprovides data analysis workflow templates compatible with thesystemPipeRsoftware package (H Backman and Girke 2016). The latter is a Workflow Management System (WMS) for designing and running end-to-end analysis workflows with automated report generation for a wide range of data analysis applications. Support for running external software is provided by a command-line interface (CLI) that adopts the Common Workflow Language (CWL). How to use systemPipeRis explained in its main vignettehere. The workflow templates provided by systemPipeRdata come equipped with sample data and the necessary parameter files required to run a selected workflow. This setup simplifies the learning process of using systemPipeR, facilitates testing of workflows, and serves as a foundation for designing new workflows. The standardized directory structure (Figure 1) utilized by the workflow templates and their sample data is outlined in the Directory Structuresection of systemPipeR's main vignette.

Figure 1: Directory structure ofsystemPipeR's workflows. For details, see here.

Getting started

Installation

The systemPipeRdata package is available at Bioconductor and can be installed from within R as follows.

if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager")
BiocManager::install("systemPipeRdata")

Loading package and documentation

library("systemPipeRdata")  # Loads the package

library(help = "systemPipeRdata")  # Lists package info
vignette("systemPipeRdata")  # Opens vignette

Overview of workflow templates

An overview table of workflow templates, included in systemPipeRdata, can be returned as shown below. By clicking the URLs in the last column of the below workflow list, users can view the Rmd source file of a workflow, as well as the final HTML report generated after running a workflow on the provided test data. A list of the default data analysis steps included in each workflow is given here. Additional workflow templates are available on this project’s GitHub organization (for details, seebelow). To create an empty workflow template without any test data included, users want to choose the new template, which includes only the required directory structure and parameter files.

availableWF()

Table 1: Workflow templates

Load a workflow

The chosen example below uses the genWorkenvir function from thesystemPipeRdata package to create an RNA-Seq workflow environment (selected under workflow="rnaseq") that is fully populated with a small test data set, including FASTQ files, reference genome and annotation data. The name of the resulting workflow directory can be specified under the mydirname argument. The default NULL uses the name of the chosen workflow. An error is issued if a directory of the same name and path exists already. After this, the user’s R session needs to be directed into the resulting rnaseq directory (here withsetwd). The other workflow templates from the abovetable can be loaded the same way.

library(systemPipeRdata)
genWorkenvir(workflow = "rnaseq")
setwd("rnaseq")

On Linux and OS X systems the same can be achieved from the command-line of a terminal with the following commands.

$ Rscript -e "systemPipeRdata::genWorkenvir(workflow='rnaseq', mydirname='rnaseq')"
$ cd rnaseq

Run and visualize workflow

For running and working with systemPipeR workflows, users want to visitsystemPipeR’s main vignette. The following gives only a very brief preview on how to run workflows, and create scientific and technical reports.

After a workflow environment (directory) has been created and the corresponding R session directed into the resulting directory (here rnaseq), the workflow can be loaded from the included R Markdown file (Rmd, here systemPipeRNAseq.Rmd). This template provides common data analysis steps that are typical for RNA-Seq workflows. Users have the options to add, remove, modify workflow steps by applying these changes to the sal workflow management container directly, or updating the Rmd file first and then updating sal accordingly.

library(systemPipeR)
sal <- SPRproject()
sal <- importWF(sal, file_path = "systemPipeRNAseq.Rmd", verbose = FALSE)

The default analysis steps of the imported RNA-Seq workflow are listed below. Users can modify the existing steps, add new ones or remove steps as needed.

Default analysis steps in RNA-Seq Workflow

Read preprocessing
- Quality filtering (trimming)
- FASTQ quality report
Alignments: HISAT2 (or any other RNA-Seq aligner)
Alignment stats
Read counting
Sample-wise correlation analysis
Analysis of differentially expressed genes (DEGs)
GO term enrichment analysis
Gene-wise clustering

Once the workflow has been loaded into sal, it can be executed from start to finish (or partially) with the runWF command. However, running the workflow will only be possible if all dependent CL software is installed on a user’s system. Their names and availability on a system can be listed withlistCmdTools(sal, check_path=TRUE).

sal <- runWF(sal)

Workflows can be visualized as topology graphs using the plotWF function.

plotWF(sal)

Figure 1: Toplogy graph of RNA-Seq workflow

Scientific and technical reports can be generated with the renderReport andrenderLogs functions, respectively. Scientific reports can also be generated with the render function of the rmarkdown package. The technical reports are based on log information that systemPipeR collects during workflow runs.

# Scietific report
sal <- renderReport(sal)
rmarkdown::render("systemPipeRNAseq.Rmd", clean = TRUE, output_format = "BiocStyle::html_document")

# Technical (log) report
sal <- renderLogs(sal)

Additional workflow templates

The project’s GitHub Organization hosts a repository of workflow templates, containing both well-established and experimental workflows. Within the R environment, the same availableWF function mentioned earlier can be utilized to retrieve a list of the workflows in this collection.

availableWF(github = TRUE)

To download these workflow templates, users can either run the below git clone command from a terminal, or visit the corresponding GitHub page of a chosen workflow via the provided URLs, and then download it as a Zip file and uncompress it. Note, the following lines of code need to be run from a terminal (not R console, e.g. terminal in RStudio) on a system where the git software is installed.

$ git clone <...> # Provide under <...> URL of chosen workflow from table above.
$ cd <Workflow Name>

After a workflow template has been downloaded, one can run it the same way as outlined above.

Useful functionalities

Create workflow templates interactively

It is possible to create a new workflow environment from RStudio. This can be done by selecting File -> New File -> R Markdown -> From Template -> systemPipeR New WorkFlow. This option creates a template workflow that has the expected directory structure (see here).

Figure 2: Selecting workflow template within RStudio.

Return paths to sample data

The paths to the sample data provided by the systemPipeRdata package can be returned with the the pathList function.

pathList()[1:2]

## $targets
## [1] "/tmp/Rtmpc5HYvB/Rinsta48133c46495c/systemPipeRdata/extdata/param/targets.txt"
## 
## $targetsPE
## [1] "/tmp/Rtmpc5HYvB/Rinsta48133c46495c/systemPipeRdata/extdata/param/targetsPE.txt"

Analysis steps in selected workflows

The following gives an overview of the default data analysis steps used by selected workflow templates included in the systemPipeRdata package (see Table 1). The workflows hosted on this project’s GitHub Organization are not considered here.

Any of the workflows included below can be loaded by assigning their name to the workflow argument of the genWorkenvir function. The workflow names can be looked up under the ‘Name’ column of Table 1.

library(systemPipeRdata)
genWorkenvir(workflow = "...")

Generic template

[ Vignette ]

This Generic workflow (named new) is intended to be used as a template for creating new workflows from scratch where users can add steps by copying and pasting existing R or CL steps as needed, and populate them with their own code. In its current form, this mini workflow will export a test dataset to multiple files, compress/decompress the exported files, import them back into R, and then perform a simple statistical analysis and plot the results.

R step: export tabular data to files
CL step: compress files
CL step: uncompress files
R step: import files and plot summary statistics

RNA-Seq workflow

[ Vignette ]

Read preprocessing
- Quality filtering (trimming)
- FASTQ quality report
Alignments: HISAT2 (or any other RNA-Seq aligner)
Alignment stats
Read counting
Sample-wise correlation analysis
Analysis of differentially expressed genes (DEGs)
GO term enrichment analysis
Gene-wise clustering

ChIP-Seq Workflow

[ Vignette ]

Read preprocessing
- Quality filtering (trimming)
- FASTQ quality report
Alignments: Bowtie2 or rsubread
Alignment stats
Peak calling: MACS2
Peak annotation with genomic context
Differential binding analysis
GO term enrichment analysis
Motif analysis

VAR-Seq Workflow

[ Vignette ]

Read preprocessing +Quality filtering (trimming) +FASTQ quality report
Alignments: bwa or other
Variant calling: GATK, BCFtools
Variant filtering: VariantTools and VariantAnnotation
Variant annotation: VariantAnnotation
Combine results from many samples
Summary statistics of samp

Ribo-Seq Workflow

[ Vignette ]

Read preprocessing
- Adaptor trimming and quality filtering
- FASTQ quality report
Alignments: HISAT2 (or any other RNA-Seq aligner)
Alignment stats
Compute read distribution across genomic features
Adding custom features to workflow (e.g. uORFs)
Genomic read coverage along transcripts
Read counting
Sample-wise correlation analysis
Analysis of differentially expressed genes (DEGs)
GO term enrichment analysis
Gene-wise clustering
Differential ribosome binding (translational efficiency)

scRNA-Seq Workflow

[ Vignette ]

Import of single cell read count data
Basic stats on input data
QC of cell count data
Cell filtering
Normalization
Identify high variable genes
Scaling
Embedding with tSNE, UMAP, and PCA
Cell clustering and marker gene classification
Cell type classification
Co-visualizatioin of cell types and clusters

BLAST Workflow

[ Vignette ]

Load query sequences
Select and prepare BLASTable databases
Run BLAST against different databases

Cheminformatics Workflow

[ Vignette ]

Import small molecules stored in SDF file
Visualize small molecule structures
Create atom pair and finger print databases for structure similarity searching
Compute all-against-all structural similarities
Hierarchical clustering and PCA of structural similarities
Plot heat map

Version information

sessionInfo()

## R version 4.5.0 beta (2025-04-02 r88102)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.2 LTS
## 
## Matrix products: default
## BLAS:   /home/biocbuild/bbs-3.22-bioc/R/lib/libRblas.so 
## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.12.0  LAPACK version 3.12.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_GB              LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## time zone: America/New_York
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] stats4    stats     graphics  grDevices utils    
## [6] datasets  methods   base     
## 
## other attached packages:
##  [1] magrittr_2.0.3              systemPipeRdata_2.13.0     
##  [3] systemPipeR_2.15.0          ShortRead_1.67.0           
##  [5] GenomicAlignments_1.45.0    SummarizedExperiment_1.39.0
##  [7] Biobase_2.69.0              MatrixGenerics_1.21.0      
##  [9] matrixStats_1.5.0           BiocParallel_1.43.0        
## [11] Rsamtools_2.25.0            Biostrings_2.77.0          
## [13] XVector_0.49.0              GenomicRanges_1.61.0       
## [15] GenomeInfoDb_1.45.0         IRanges_2.43.0             
## [17] S4Vectors_0.47.0            BiocGenerics_0.55.0        
## [19] generics_0.1.3              BiocStyle_2.37.0           
## 
## loaded via a namespace (and not attached):
##  [1] gtable_0.3.6            xfun_0.52              
##  [3] bslib_0.9.0             hwriter_1.3.2.1        
##  [5] ggplot2_3.5.2           remotes_2.5.0          
##  [7] htmlwidgets_1.6.4       latticeExtra_0.6-30    
##  [9] lattice_0.22-7          vctrs_0.6.5            
## [11] tools_4.5.0             bitops_1.0-9           
## [13] parallel_4.5.0          tibble_3.2.1           
## [15] pkgconfig_2.0.3         Matrix_1.7-3           
## [17] RColorBrewer_1.1-3      lifecycle_1.0.4        
## [19] GenomeInfoDbData_1.2.14 stringr_1.5.1          
## [21] compiler_4.5.0          deldir_2.0-4           
## [23] munsell_0.5.1           codetools_0.2-20       
## [25] htmltools_0.5.8.1       sass_0.4.10            
## [27] yaml_2.3.10             pillar_1.10.2          
## [29] crayon_1.5.3            jquerylib_0.1.4        
## [31] DelayedArray_0.35.0     cachem_1.1.0           
## [33] abind_1.4-8             tidyselect_1.2.1       
## [35] digest_0.6.37           stringi_1.8.7          
## [37] dplyr_1.1.4             bookdown_0.43          
## [39] fastmap_1.2.0           grid_4.5.0             
## [41] colorspace_2.1-1        cli_3.6.4              
## [43] SparseArray_1.9.0       S4Arrays_1.9.0         
## [45] UCSC.utils_1.5.0        scales_1.3.0           
## [47] rmarkdown_2.29          pwalign_1.5.0          
## [49] httr_1.4.7              jpeg_0.1-11            
## [51] interp_1.1-6            png_0.1-8              
## [53] kableExtra_1.4.0        evaluate_1.0.3         
## [55] knitr_1.50              viridisLite_0.4.2      
## [57] rlang_1.1.6             Rcpp_1.0.14            
## [59] glue_1.8.0              xml2_1.3.8             
## [61] BiocManager_1.30.25     formatR_1.14           
## [63] svglite_2.1.3           rstudioapi_0.17.1      
## [65] jsonlite_2.0.0          R6_2.6.1               
## [67] systemfonts_1.2.2

Funding

This project was supported by funds from the National Institutes of Health (NIH) and the National Science Foundation (NSF).

systemPipeRdata: Workflow templates and sample data (original) (raw)

Introduction

Getting started

Installation

Loading package and documentation

Overview of workflow templates

Load a workflow

Run and visualize workflow

Additional workflow templates

Useful functionalities

Create workflow templates interactively

Return paths to sample data

Analysis steps in selected workflows

Generic template

RNA-Seq workflow

ChIP-Seq Workflow

VAR-Seq Workflow

Ribo-Seq Workflow

scRNA-Seq Workflow

BLAST Workflow

Cheminformatics Workflow

Version information

Funding

References