pipen (original) (raw)
Project description
A pipeline framework for python
Documentation | ChangeLog | Examples | API
Features
- Easy to use
- Nearly zero-configuration
- Nice logging
- Highly extendable
Installation
pip install -U pipen
Quickstart
example.py
from pipen import Proc, Pipen, run
class P1(Proc): """Sort input file""" input = "infile" input_data = ["/tmp/data.txt"] output = "outfile:file:intermediate.txt" script = "cat {{in.infile}} | sort > {{out.outfile}}"
class P2(Proc): """Paste line number""" requires = P1 input = "infile" output = "outfile:file:result.txt" script = "paste <(seq 1 3) {{in.infile}} > {{out.outfile}}"
class MyPipeline(Pipen):
starts = P1
if name == "main": # MyPipeline().run() run("MyPipeline", starts=P1)
echo -e "3\n2\n1" > /tmp/data.txt python example.py
06-09 23:15:29 I core _____________________________________ __ 06-09 23:15:29 I core ___ __ ___ /_ __ __ / | / / 06-09 23:15:29 I core __ // /_ / __ // / / __ |/ / 06-09 23:15:29 I core _ // / _ / / _ /| / 06-09 23:15:29 I core // /__/ // /_/ // |_/ 06-09 23:15:29 I core 06-09 23:15:29 I core version: 0.15.3 06-09 23:15:29 I core 06-09 23:15:29 I core ╔══════════════════════════════════════════════════════════════════╗ 06-09 23:15:29 I core ║ MYPIPELINE ║ 06-09 23:15:29 I core ╚══════════════════════════════════════════════════════════════════╝ 06-09 23:15:29 I core plugins : verbose v0.12.0 06-09 23:15:29 I core # procs : 2 06-09 23:15:29 I core profile : default 06-09 23:15:29 I core outdir : /home/pwwang/github/pipen/MyPipeline-output 06-09 23:15:29 I core cache : True 06-09 23:15:29 I core dirsig : 1 06-09 23:15:29 I core error_strategy : ignore 06-09 23:15:29 I core forks : 1 06-09 23:15:29 I core lang : bash 06-09 23:15:29 I core loglevel : info 06-09 23:15:29 I core num_retries : 3 06-09 23:15:29 I core scheduler : local 06-09 23:15:29 I core submission_batch: 8 06-09 23:15:29 I core template : liquid 06-09 23:15:29 I core workdir : /home/pwwang/github/pipen/.pipen/MyPipeline 06-09 23:15:29 I core plugin_opts : 06-09 23:15:29 I core template_opts : 06-09 23:15:31 I core 06-09 23:15:31 I core ╭─────────────────────────────── P1 ───────────────────────────────╮ 06-09 23:15:31 I core │ Sort input file │ 06-09 23:15:31 I core ╰──────────────────────────────────────────────────────────────────╯ 06-09 23:15:31 I core P1: Workdir: '/home/pwwang/github/pipen/.pipen/MyPipeline/P1' 06-09 23:15:31 I core P1: <<< [START] 06-09 23:15:31 I core P1: >>> ['P2'] 06-09 23:15:31 I verbose P1: size: 1 06-09 23:15:31 I verbose P1: [0/0] in.infile: /tmp/data.txt 06-09 23:15:31 I verbose P1: [0/0] out.outfile: /home/pwwang/github/pipen/.pipen/MyPipeline/P1/0/output/intermediate.txt 06-09 23:15:33 I verbose P1: Time elapsed: 00:00:02.018s 06-09 23:15:33 I core 06-09 23:15:33 I core ╭═══════════════════════════════ P2 ═══════════════════════════════╮ 06-09 23:15:33 I core ║ Paste line number ║ 06-09 23:15:33 I core ╰══════════════════════════════════════════════════════════════════╯ 06-09 23:15:33 I core P2: Workdir: '/home/pwwang/github/pipen/.pipen/MyPipeline/P2' 06-09 23:15:33 I core P2: <<< ['P1'] 06-09 23:15:33 I core P2: >>> [END] 06-09 23:15:33 I verbose P2: size: 1 06-09 23:15:33 I verbose P2: [0/0] in.infile: /home/pwwang/github/pipen/.pipen/MyPipeline/P1/0/output/intermediate.txt 06-09 23:15:33 I verbose P2: [0/0] out.outfile: /home/pwwang/github/pipen/MyPipeline-output/P2/result.txt 06-09 23:15:35 I verbose P2: Time elapsed: 00:00:02.009s 06-09 23:15:35 I core
MYPIPELINE: 100%|█████████████████████████████| 2/2 [00:06<00:00, 0.36 procs/s]cat ./MyPipeline-output/P2/result.txt 1 1 2 2 3 3
Examples
See more examples at examples/ and a more realcase example at:
https://github.com/pwwang/pipen-report/tree/master/example
Plugin gallery
Plugins make pipen even better.
- pipen-annotate: Use docstring to annotate pipen processes
- pipen-args: Command line argument parser for pipen
- pipen-board: Visualize configuration and running of pipen pipelines on the web
- pipen-diagram: Draw pipeline diagrams for pipen
- pipen-dry: Dry runner for pipen pipelines
- pipen-filters: Add a set of useful filters for pipen templates.
- pipen-lock: Process lock for pipen to prevent multiple runs at the same time.
- pipen-log2file: Save running logs to file for pipen
- pipen-poplog: Populate logs from jobs to running log of the pipeline
- pipen-report: Generate report for pipen
- pipen-runinfo: Save running information to file for pipen
- pipen-verbose: Add verbosal information in logs for pipen.
- pipen-gcs: A plugin for pipen to handle files in Google Cloud Storage.
- pipen-cli-init: A pipen CLI plugin to create a pipen project (pipeline)
- pipen-cli-ref: Make reference documentation for processes
- pipen-cli-require: A pipen cli plugin check the requirements of a pipeline
- pipen-cli-run: A pipen cli plugin to run a process or a pipeline
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.