GitHub - hammerlab/ketrew: Keep Track of Experimental Workflows (original) (raw)
Ketrew: Keep Track of Experimental Workflows
Ketrew is:
- An OCaml library providing an EDSL API to define complex and convoluted workflows (interdependent steps/programs using a lot of data, with many parameter variations, running on different hosts with various schedulers).
- A client-server application to interact with these workflows. The engine at heart of the server takes care of orchestrating workflows, and keeps track of everything that succeeds, fails, or gets lost.
See also thedocumentationfor various releases.
If you have any questions, you may submit anissue, or join the authors on the public “Slack” channel of the Hammer Lab:
Build & Install
See the specific documentation on building and installing.TL;DR for OCaml hackers:
opam switch 4.03.0
opam install [postgresql] [tls] ketrew
Getting Started
Ketrew is very flexible and hence may seem difficult to understand and setup at first. Let's get a minimal setup ready and a workflow running on it.
Server-side Setup
This tutorial requiresdockerand docker-compose:
alias kdc='./tools/docker-compose/kdc.sh'
kdc config --services
Let's get a Ketrew server with a PostgreSQL database and a Coclobas scheduler talking to each other:
Check that everything is running:
Check the Ketrew server status:
curl http://127.0.0.1:8123/hello || echo 'Not ready'
After a minute or two you can check that everything is setup by visiting the Ketrew UI: http://127.0.0.1:8123/gui?token=nekot:
At any moment you can take everything down with:
Or use other inspection commands:
Client
We can now create a Ketrew client configuration, please choose a directory:
export KETREW_ROOT=$HOME/tmp/kclient-config/
and initialize Ketrew there:
ketrew init --configuration-path $KETREW_ROOT \
--just-client http://127.0.0.1:8123/gui?token=nekot
The ketrew submit
sub-command can create one-command workflows (uses the$KETREW_ROOT
path):
ketrew submit \
--wet-run --tag 1st-workflow --tag command-line \
--daemonize /tmp/KT,'du -sh $HOME'
The job will appear on the WebUI and you can inspect/restart/kill it.
If you don't like Web UI's you can use the text-based UI:
$ ketrew interact
[ketrew]
Main menu
Press a single key:
* [q]: Quit
* [v]: Toggle verbose
* [s]: Display current status
* [l]: Loop displaying the status
* [k]: Kill targets
* [e]: The Target Explorer™
As you can see, just from the command line, you can use ketrew submit
to launch daemonized tasks. To go further we need to use Ketrew's EDSL.
The EDSL: Defining Workflows
Overview
The EDSL is an OCaml library where functions are used to build a workflow data-structure. Ketrew.Client.submit_workflow
is used to submit that datastructure to the engine.
A workflow is a graph of “workflow-nodes” (sometimes called “targets”).
There are three kinds of links (edges) between nodes:
depends_on
: nodes that need to be ensured or satisfied before a node can start,on_failure_activate
: nodes that will be activated if the node fails, andon_success_activate
: nodes that will be activated only after a node succeeds.
See the Ketrew.EDSL.workflow_node
function documentation for details. Any OCaml program can use the EDSL (script, compiled, or even inside the toplevel). See the documentation of the EDSL API (Ketrew.EDSL
).
A Quick Taste
You can run these commands for example in utop
(opam install utop
).
Load the Ketrew library to build and submit workflows:
#use "topfind";; #thread;; #require "ketrew";;
Globally tell the Ketrew client to get its configuration from the file createdabove:
let () = Unix.putenv "KETREW_CONFIGURATION" (Sys.getenv "HOME" ^ "/tmp/kclient-config/configuration.ml");;
Submit an “empty” workflow-node to Ketrew (i.e. a node that does not do nor_ensure_ anything):
let () = let open Ketrew.EDSL in workflow_node without_product ~name:"Mostly Empty Node" |> Ketrew.Client.submit_workflow;;
Submit a node mostly equivalent to the one submitted from the command-line (ketrew submit --daemonize ...
):
let () = let open Ketrew.EDSL in workflow_node without_product ~name:"Equivalent to the Command Line one" ~make:( daemonize ~using:`Python_daemon ~host:Host.(parse "/tmp/KT") Program.(sh "du -sh $HOME") ) ~tags:["not-1st-workflow"; "not-command-line"] |> Ketrew.Client.submit_workflow;;
Run a command, in a docker container scheduler by the Coclobas server (also setup by docker-compose
above):
#require "coclobas.ketrew_backend";;
let () = let open Ketrew.EDSL in workflow_node without_product ~name:"Uses a docker image to run some commands" ~make:( (* We use a different “backend”: ) Coclobas_ketrew_backend.Plugin.local_docker_program ~tmp_dir:"/tmp/secotrec-local-shared-temp" ~base_url:"http://coclo:8082" ~image:"ubuntu" Program.( ( sh "sudo mkdir -m 777 -p /cloco-kube/playground" && *) sh "echo User" && sh "whoami" && sh "echo Host" && sh "hostname" && sh "echo Machine" && sh "uname -a" && exec ["sleep"; "42"] ) ) ~tags:["using-coclobas"; "from-utop"] |> Ketrew.Client.submit_workflow;;
The Ketrew WebUI should look like this:
Bigger Example
The following script extends the previous examples with the capability to send emails upon the success or failure of your command.
#use "topfind" #thread #require "ketrew"
let run_command_with_daemonize ~cmd ~email =
let module KEDSL = Ketrew.EDSL in
(* Where to run stuff: )
let host = KEDSL.Host.tmp_on_localhost in
( A node that Ketrew will activate after cmd completes,
on its success or failure. *)
let email_target ~success =
let msg_string = if success then "succeeded" else "failed" in
let e_program =
KEDSL.Program.shf "echo "'%s' %s" | mail -s "Status update" %s"
cmd msg_string
email
in
let e_process =
KEDSL.daemonize ~using:Python_daemon ~host e_program in KEDSL.workflow_node KEDSL.without_product ~name:("email result " ^ msg_string) ~make:e_process in (* The function
KEDSL.workflow_nodecreates a node in the workflow graph. The value
KEDSL.without_productmeans this node does not “produce” anything, it is like a
.PHONYtarget in
make. *) KEDSL.workflow_node KEDSL.without_product ~name:"daemonized command with email notification" ~make:( (* A “program” is a datastructure representing an “extended shell script”. *) let program = KEDSL.Program.sh cmd in KEDSL.daemonize ~host ~using:
Python_daemon program
)
~edges:[
KEDSL.on_success_activate (email_target true);
KEDSL.on_failure_activate (email_target false);
]
let () =
(* Grab the command line arguments. )
let cmd = Sys.argv.(1) in
let email = Sys.argv.(2) in
( Create the workflow with the first argument of the command line: )
let workflow = run_command_with_daemonize ~cmd ~email in
( Then, Client.submit_workflow
is the only function that “does”
something, it submits the constructed workflow to the engine: *)
Ketrew.Client.submit_workflow workflow
You can run this script from the shell with
export KETREW_CONFIGURATION=$HOME/kclient-config/configuration.ml
ocaml daemonize_workflow.ml 'du -sh $HOME' myaddress@email.com
Checking in with the GUI, we'll have a few new nodes, here is an example of execution where the daemonized program does not know about the mail
command:
Where to Go Next
From here:
- To write workflows for Ketrew see:
- examples of more and more complicated workflows: src/test/Workflow_Examples.ml for examples
- the documentation of the
Ketrew.EDSL
API.
- To configure Ketrew use the configuration file documentation.
- You may want to “extend” Ketrew with new ways of running “long-running" computations: see the documentation on plugins, and the examples in the library: like Ketrew.Lsf or in the tests: src/test/dummy_plugin.ml.
- You may want to extend Ketrew, or preconfigure it, without configuration files or dynamically loaded libraries: just create your own comand-line app.
- If you are using Ketrew in server mode, you may want to know about the commands that the server can understand as it listens on a Unix-pipe.
- You may want to call out directly to the HTTP API(i.e. without
ketrew
as a client). - If you want to help or simply to understand Ketrew see the development documentation, and have a look at the modules like Ketrew.Engine.
License
It's Apache 2.0.