GitHub - k1LoW/runn: runn is a package/tool for running operations following a scenario. (original) (raw)

runn

build Coverage Code to Test Ratio Test Execution Time

runn ( means "Run N". is pronounced /rʌ́n én/. ) is a package/tool for running operations following a scenario.

Key features of runn are:

Online book

Quickstart

You can use the runn new command to quickly start creating scenarios (runbooks).

🚀 Create and run scenario using curl or grpcurl commands:

docs/runn.svg

Command details

$ curl https://httpbin.org/json -H "accept: application/json" { "slideshow": { "author": "Yours Truly", "date": "date of publication", "slides": [ { "title": "Wake up to WonderWidgets!", "type": "all" }, { "items": [ "Why WonderWidgets are great", "Who buys WonderWidgets" ], "title": "Overview", "type": "all" } ], "title": "Sample Slide Show" } } $ runn new --and-run --desc 'httpbin.org GET' --out http.yml -- curl https://httpbin.org/json -H "accept: application/json" $ grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello { "reply": "hello alice" } $ runn new --and-run --desc 'grpcb.in Call' --out grpc.yml -- grpcurl -d '{"greeting": "alice"}' grpcb.in:9001 hello.HelloService/SayHello $ runn list *.yml Desc Path If

grpcb.in Call grpc.yml httpbin.org GET http.yml $ runn run *.yml ..

2 scenarios, 0 skipped, 0 failures

🚀 Create scenario using access log:

docs/runn_axslog.svg

Command details

$ cat access_log 183.87.255.54 - - [18/May/2019:05:37:09 +0200] "GET /?post=%3script%3ealert(1); HTTP/1.0" 200 42433 62.109.16.162 - - [18/May/2019:05:37:12 +0200] "GET /core/files/js/editor.js/?form=\xeb\x2a\x5e\x89\x76\x08\xc6\x46\x07\x00\xc7\x46\x0c\x00\x00\x00\x80\xe8\xdc\xff\xff\xff/bin/sh HTTP/1.0" 200 81956 87.251.81.179 - - [18/May/2019:05:37:13 +0200] "GET /login.php/?user=admin&amount=100000 HTTP/1.0" 400 4797 103.36.79.144 - - [18/May/2019:05:37:14 +0200] "GET /authorize.php/.well-known/assetlinks.json HTTP/1.0" 200 9436 $ cat access_log| runn new --out axslog.yml $ cat axslog.yml| yq desc: Generated by runn new runners: req: https://dummy.example.com steps:

$

Usage

runn can run a multi-step scenario following a runbook written in YAML format.

As a tool for scenario based testing / As a tool for automation.

runn can run one or more runbooks as a CLI tool.

$ runn list path/to/**/*.yml id: desc: if: steps: path

a1b7b02 Only if included included 2 p/t/only_if_included.yml 85ccd5f List projects. 4 p/t/p/list.yml 47d7ef7 List users. 3 p/t/u/list.yml 97f9884 Login 2 p/t/u/login.yml 2249d1b Logout 3 p/t/u/logout.yml $ runn run path/to/**/*.yml S....

5 scenarios, 1 skipped, 0 failures

As a test helper package for the Go language.

runn can also behave as a test helper for the Go language.

Run N runbooks using httptest.Server and sql.DB

func TestRouter(t testing.T) { ctx := context.Background() dsn := "username:password@tcp(localhost:3306)/testdb" db, err := sql.Open("mysql", dsn) if err != nil { log.Fatal(err) } dbr, err := sql.Open("mysql", dsn) if err != nil { log.Fatal(err) } ts := httptest.NewServer(NewRouter(db)) t.Cleanup(func() { ts.Close() db.Close() dbr.Close() }) opts := []runn.Option{ runn.T(t), runn.Runner("req", ts.URL), runn.DBRunner("db", dbr), } o, err := runn.Load("testdata/books/**/.yml", opts...) if err != nil { t.Fatal(err) } if err := o.RunN(ctx); err != nil { t.Fatal(err) } }

Run single runbook using httptest.Server and sql.DB

func TestRouter(t *testing.T) { ctx := context.Background() dsn := "username:password@tcp(localhost:3306)/testdb" db, err := sql.Open("mysql", dsn) if err != nil { log.Fatal(err) } dbr, err := sql.Open("mysql", dsn) if err != nil { log.Fatal(err) } ts := httptest.NewServer(NewRouter(db)) t.Cleanup(func() { ts.Close() db.Close() dbr.Close() }) opts := []runn.Option{ runn.T(t), runn.Book("testdata/books/login.yml"), runn.Runner("req", ts.URL), runn.DBRunner("db", dbr), } o, err := runn.New(opts...) if err != nil { t.Fatal(err) } if err := o.Run(ctx); err != nil { t.Fatal(err) } }

Run N runbooks using grpc.Server

func TestServer(t testing.T) { addr := "127.0.0.1:8080" l, err := net.Listen("tcp", addr) if err != nil { t.Fatal(err) } ts := grpc.NewServer() myapppb.RegisterMyappServiceServer(s, NewMyappServer()) reflection.Register(s) go func() { ts.Serve(l) }() t.Cleanup(func() { ts.GracefulStop() }) opts := []runn.Option{ runn.T(t), runn.Runner("greq", fmt.Sprintf("grpc://%s", addr), } o, err := runn.Load("testdata/books/**/.yml", opts...) if err != nil { t.Fatal(err) } if err := o.RunN(ctx); err != nil { t.Fatal(err) } }

Run N runbooks with http.Handler and sql.DB

func TestRouter(t testing.T) { ctx := context.Background() dsn := "username:password@tcp(localhost:3306)/testdb" db, err := sql.Open("mysql", dsn) if err != nil { log.Fatal(err) } dbr, err := sql.Open("mysql", dsn) if err != nil { log.Fatal(err) } t.Cleanup(func() { db.Close() dbr.Close() }) opts := []runn.Option{ runn.T(t), runn.HTTPRunnerWithHandler("req", NewRouter(db)), runn.DBRunner("db", dbr), } o, err := runn.Load("testdata/books/**/.yml", opts...) if err != nil { t.Fatal(err) } if err := o.RunN(ctx); err != nil { t.Fatal(err) } }

Examples

See the details

Runbook ( runn scenario file )

The runbook file has the following format.

step: section accepts list or ordered map.

List:

desc: Login and get projects. runners: req: https://example.com/api/v1 db: mysql://root:mypass@localhost:3306/testdb vars: username: alice password: ${TEST_PASS} steps:

db:
  query: SELECT * FROM users WHERE name = '{{ vars.username }}'

Map:

desc: Login and get projects. runners: req: https://example.com/api/v1 db: mysql://root:mypass@localhost:3306/testdb vars: username: alice password: ${TEST_PASS} steps: find_user: db: query: SELECT * FROM users WHERE name = '{{ vars.username }}' login: req: /login: post: body: application/json: email: "{{ steps.find_user.rows[0].email }}" password: "{{ vars.password }}" test: steps.login.res.status == 200 list_projects: req: /projects: get: headers: Authorization: "token {{ steps.login.res.body.session_token }}" body: null test: steps.list_projects.res.status == 200 count_projects: test: len(steps.list_projects.res.body.projects) > 0

List:

color

Map:

color

desc:

Description of runbook.

desc: Login and get projects. runners: req: https://example.com/api/v1 vars: username: alice steps: [...]

labels:

Labels of runbook.

desc: Login runners: req: https://example.com/api/v1 labels:

Runbooks to be run can be filtered by labels.

$ runn run path/to/**/*.yml --label users --label projects

$ runn run path/to/**/*.yml --label 'users and auth'

runners:

Mapping of runners that run steps: of runbook.

In the steps: section, call the runner with the key specified in the runners: section.

Built-in runners such as test runner do not need to be specified in this section.

runners: ghapi: ${GITHUB_API_ENDPOINT} idp: https://auth.example.com db: my:dbuser:${DB_PASS}@hostname:3306/dbname

In the example, each runner can be called by ghapi:, idp: or db: in steps:.

hostRules:

Allows remapping any request hostname to another hostname, IP address in HTTP/gRPC/DB/CDP/SSH runners.

hostRules: example.com: 127.0.0.1:8080 '*.example.test': 192.168.0.16

vars:

Mapping of variables available in the steps: of runbook.

vars: username: alice@example.com token: ${SECRET_TOKEN}

In the example, each variable can be used in {{ vars.username }} or {{ vars.token }} in steps:.

secrets:

List of secret var names to be masked.

secrets:

debug:

Enable debug output for runn.

interval:

Interval between steps.

if:

Conditions for skip all steps.

if: included # Run steps only if included

skipTest:

Skip all test: sections

force:

Force all steps to run.

trace:

Add tokens for tracing to headers and queries by default.

Currently, HTTP runner, gRPC runner and DB runner are supported.

loop:

Loop setting for runbook.

Simple loop runbook

or

loop: count: 10 steps: [...]

Retry runbook

It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed.

loop: count: 10 until: 'outcome == "success"' # until the runbook outcome is successful. minInterval: 0.5 # sec maxInterval: 10 # sec

jitter: 0.0

interval: 5

multiplier: 1.5

steps: waitingroom: req: /cart/in: post: body: [...]

concurrency:

Runbooks with the same key are assured of a single run at the same time.

concurrency: use-shared-db

or

concurrency:

needs:

It is possible to identify runbooks that must be pre-run.

needs: prebook: path/to/prebook.yml prebook2: path/to/prebook2.yml

Values bound by the bind runner can be referenced by needs.<key>. *.

steps:

Steps to run in runbook.

The steps are invoked in order from top to bottom.

Any return values are recorded for each step.

When steps: is array, recorded values can be retrieved with {{ steps[*].* }}.

steps:

db:
  query: SELECT * FROM users WHERE name = '{{ vars.username }}'

When steps: is map, recorded values can be retrieved with {{ steps.<key>.* }}.

steps: find_user: db: query: SELECT * FROM users WHERE name = '{{ vars.username }}' user_info: req: /users/{{ steps.find_user.rows[0].id }}: get: body: null

steps[*].desc: steps.<key>.desc:

Description of step.

steps:

desc: Login
req:
  /login:
    post:
      body:

[...]

steps[*].if: steps.<key>.if:

Conditions for skip step.

steps: login: if: 'len(vars.token) == 0' # Run step only if var.token is not set req: /login: post: body: [...]

steps[*].loop: steps.<key>.loop:

Loop settings for steps.

Simple loop step

steps: multicartin: loop: 10 req: /cart/in: post: body: application/json: product_id: "{{ i }}" # The loop count (0..9) is assigned to i. [...]

or

steps: multicartin: loop: count: 10 req: /cart/in: post: body: application/json: product_id: "{{ i }}" # The loop count (0..9) is assigned to i. [...]

Retry step

It can be used as a retry mechanism by setting a condition in the until: section.

If the condition of until: is met, the loop is broken without waiting for the number of count: to be run.

Also, if the run of the number of count: completes but does not satisfy the condition of until:, then the step is considered to be failed.

steps: waitingroom: loop: count: 10 until: 'steps.waitingroom.res.status == "201"' # Store values of latest loop minInterval: 500ms maxInterval: 10 # sec # jitter: 0.0 # interval: 5 # multiplier: 1.5 req: /cart/in: post: body: [...]

Variables to be stored

runn can use variables and functions when running step.

Also, after step runs, HTTP responses, DB query results, etc. are automatically stored in variables.

The values are stored in predefined variables.

Variable name Description
vars Values set in the vars: section
steps Return values for each step
i Loop index (only in loop: section)
env Environment variables
current Return values of current step
previous Return values of previous step
parent Variables of parent runbook (only included)

Runner

HTTP Runner: Do HTTP request

Use https:// or http:// scheme to specify HTTP Runner.

When the step is invoked, it sends the specified HTTP Request and records the response.

runners: req: https://example.com steps:

desc: Post /users                     # description of step
req:                                  # key to identify the runner. In this case, it is HTTP Runner.
  /users:                             # path of http request
    post:                             # method of http request
      headers:                        # headers of http request
        Authorization: 'Bearer xxxxx'
      body:                           # body of http request
        application/json:             # Content-Type specification. In this case, it is "Content-Type: application/json"
          username: alice
          password: passw0rd
      trace: false                    # add `X-Runn-Trace` header to HTTP request for tracing
test: |                               # test for current step
  current.res.status == 201

See testdata/book/http.yml and testdata/book/http_multipart.yml.

Structure of recorded responses

The following response

HTTP/1.1 200 OK
Content-Length: 29
Content-Type: application/json
Date: Wed, 07 Sep 2022 06:28:20 GMT
Set-Cookie: cookie-name=cookie-value

{"data":{"username":"alice"}}

is recorded with the following structure.

status: 200                              # current.res.status
headers:
  Content-Length:
    - '29'                               # current.res.headers["Content-Length"][0]
  Content-Type:
    - 'application/json'                 # current.res.headers["Content-Type"][0]
  Date:
    - 'Wed, 07 Sep 2022 06:28:20 GMT'    # current.res.headers["Date"][0]
  Set-Cookie:
    - 'cookie-name=cookie-value'         # current.res.headers["Set-Cookie"][0]
cookies:
  cookie-name: *http.Cookie              # current.res.cookies["cookie-name"].Value
body:
  data:
    username: 'alice'                    # current.res.body.data.username
rawBody: '{"data":{"username":"alice"}}' # current.res.rawBody

Do not follow redirect

The HTTP Runner interprets HTTP responses and automatically redirects. To disable this, set notFollowRedirect to true.

runners: req: endpoint: https://example.com notFollowRedirect: true

The HTTP Runner automatically saves cookies by interpreting HTTP responses. To enable cookie sending during requests, set useCookie to true.

runners: req: endpoint: https://example.com useCookie: true

See testdata/book/cookie.yml and testdata/book/cookie_in_requests_automatically.yml.

Validation of HTTP request and HTTP response

HTTP requests sent by runn and their HTTP responses can be validated.

OpenAPI v3:

runners: myapi: endpoint: https://api.example.com openapi3: path/to/openapi.yaml # skipValidateRequest: false # skipValidateResponse: false # skipCircularReferenceCheck: false # skip checking circular references in OpenAPIv3 document.

Custom CA and Certificates

runners: myapi: endpoint: https://api.github.com cacert: path/to/cacert.pem cert: path/to/cert.pem key: path/to/key.pem # skipVerify: false

Add X-Runn-Trace header to HTTP request for tracing

runners: myapi: endpoint: https://api.github.com trace: true

gRPC Runner: Do gRPC request

Use grpc:// scheme to specify gRPC Runner.

When the step is invoked, it sends the specified gRPC Request and records the response.

runners: greq: grpc://grpc.example.com:80 steps:

desc: Request using Unary RPC                     # description of step
greq:                                             # key to identify the runner. In this case, it is gRPC Runner.
  grpctest.GrpcTestService/Hello:                 # package.Service/Method of rpc
    headers:                                      # headers of rpc
      authentication: tokenhello
    message:                                      # message of rpc
      name: alice
      num: 3
      request_time: 2022-06-25T05:24:43.861872Z
    trace: false                                  # add `x-runn-trace` header to gRPC request for tracing

runners: greq: addr: grpc.example.com:8080 tls: true cacert: path/to/cacert.pem cert: path/to/cert.pem key: path/to/key.pem # skipVerify: false # importPaths: # - protobuf/proto # protos: # - general/health.proto # - myapp/**/*.proto

See testdata/book/grpc.yml.

Structure of recorded responses

The following response

message HelloResponse { string message = 1;

int32 num = 2;

google.protobuf.Timestamp create_time = 3; }

{"create_time":"2022-06-25T05:24:43.861872Z","message":"hello","num":32}

and headers

content-type: ["application/grpc"] hello: ["this is header"]

and trailers

hello: ["this is trailer"]

are recorded with the following structure.

status: 0                                      # current.res.status
headers:
  content-type:
    - 'application/grpc'                       # current.res.headers[0].content-type
  hello:
    - 'this is header'                         # current.res.headers[0].hello
trailers:
  hello:
    - 'this is trailer'                        # current.res.trailers[0].hello
message:
  create_time: '2022-06-25T05:24:43.861872Z'   # current.res.message.create_time
  message: 'hello'                             # current.res.message.message
  num: 32                                      # current.res.message.num
messages:
  -
    create_time: '2022-06-25T05:24:43.861872Z' # current.res.messages[0].create_time
    message: 'hello'                           # current.res.messages[0].message
    num: 32                                    # current.res.messages[0].num

Add x-runn-trace header to gRPC request for tracing

runners: greq: addr: grpc.example.com:8080 trace: true

Buf

gRPC Runner supports Buf ecosystem includes Buf Schema Registry.

It can use the buf modules ( and protos ) it depends on.

runners: greq: addr: grpc.example.com:8080 bufDirs: - path/to # Set buf directories for registering buf modules and protos

runners: greq: addr: grpc.example.com:8080 bufLocks: - path/to/buf.lock # Register buf modules using buf.lock

runners: greq: addr: grpc.example.com:8080 bufConfigs: - path/to/buf.yaml # Register buf modules using buf.yaml

runners: greq: addr: grpc.example.com:8080 bufModules: - buf.build/owner/repository - buf.build/owner2/repository2

DB Runner: Query a database

Use dsn (Data Source Name) to specify DB Runner.

When step is invoked, it executes the specified query the database.

runners: db: postgres://dbuser:dbpass@hostname:5432/dbname steps:

desc: Select users            # description of step
db:                           # key to identify the runner. In this case, it is DB Runner.
  query: SELECT * FROM users; # query to execute
  trace: false                # add comment with trace token to query for tracing

See testdata/book/db.yml.

Structure of recorded responses

If the query is a SELECT clause, it records the selected rows,

-
  id: 1                           # current.rows[0].id
  username: 'alice'               # current.rows[0].username
  password: 'passw0rd'            # current.rows[0].password
  email: 'alice@example.com'      # current.rows[0].email
  created: '2017-12-05T00:00:00Z' # current.rows[0].created
-
  id: 2                           # current.rows[1].id
  username: 'bob'                 # current.rows[1].username
  password: 'passw0rd'            # current.rows[1].password
  email: 'bob@example.com'        # current.rows[1].email
  created: '2022-02-22T00:00:00Z' # current.rows[1].created

otherwise it records last_insert_id and rows_affected .

step key or current or previous: last_insert_id: 3 # current.last_insert_id rows_affected: 1 # current.rows_affected

Add comment with trace token to query for tracing

runners: db: dsn: mysql://dbuser:dbpass@hostname:3306/dbname trace: true

Support Databases

PostgreSQL:

runners: mydb: postgres://dbuser:dbpass@hostname:5432/dbname

runners: db: pg://dbuser:dbpass@hostname:5432/dbname

MySQL:

runners: testdb: mysql://dbuser:dbpass@hostname:3306/dbname

runners: db: my://dbuser:dbpass@hostname:3306/dbname

SQLite3:

runners: db: sqlite:///path/to/dbname.db

runners: local: sq://dbname.db

Cloud Spanner:

runners: testdb: spanner://test-project/test-instance/test-database

runners: db: sp://test-project/test-instance/test-database

CDP Runner: Control browser using Chrome DevTools Protocol (CDP)

Use cdp:// or chrome:// scheme to specify CDP Runner.

When the step is invoked, it controls browser via Chrome DevTools Protocol.

runners: cc: chrome://new steps:

desc: Navigate, click and get h1 using CDP  # description of step
cc:                                         # key to identify the runner. In this case, it is CDP Runner.
  actions:                                  # actions to control browser
    - navigate: https://pkg.go.dev/time
    - click: 'body > header > div.go-Header-inner > nav > div > ul > li:nth-child(2) > a'
    - waitVisible: 'body > footer'
    - text: 'h1'

See testdata/book/cdp.yml.

Functions for action to control browser

attributes (aliases: getAttributes, attrs, getAttrs)

Get the element attributes for the first element node matching the selector (sel).

actions:

record to current.attrs:

or

actions:

click

Send a mouse click event to the first element node matching the selector (sel).

actions:

or

actions:

doubleClick

Send a mouse double click event to the first element node matching the selector (sel).

actions:

or

actions:

evaluate (aliases: eval)

Evaluate the Javascript expression (expr).

actions:

or

actions:

fullHTML (aliases: getFullHTML, getHTML, html)

Get the full html of page.

actions:

record to current.html:

innerHTML (aliases: getInnerHTML)

Get the inner html of the first element node matching the selector (sel).

actions:

record to current.html:

or

actions:

latestTab (aliases: latestTarget)

Change current frame to latest tab.

localStorage (aliases: getLocalStorage)

Get localStorage items.

actions:

record to current.items:

or

actions:

location (aliases: getLocation)

Get the document location.

actions:

record to current.url:

navigate

Navigate the current frame to url page.

actions:

or

actions:

outerHTML (aliases: getOuterHTML)

Get the outer html of the first element node matching the selector (sel).

actions:

record to current.html:

or

actions:

screenshot (aliases: getScreenshot)

Take a full screenshot of the entire browser viewport.

actions:

record to current.png:

scroll (aliases: scrollIntoView)

Scroll the window to the first element node matching the selector (sel).

actions:

or

actions:

sendKeys

Send keys (value) to the first element node matching the selector (sel).

actions:

sessionStorage (aliases: getSessionStorage)

Get sessionStorage items.

actions:

record to current.items:

or

actions:

setUploadFile (aliases: setUpload)

Set upload file (path) to the first element node matching the selector (sel).

actions:

setUserAgent (aliases: setUA, ua, userAgent)

Set the default User-Agent

actions:

or

actions:

submit

Submit the parent form of the first element node matching the selector (sel).

actions:

or

actions:

text (aliases: getText)

Get the visible text of the first element node matching the selector (sel).

actions:

record to current.text:

or

textContent (aliases: getTextContent)

Get the text content of the first element node matching the selector (sel).

actions:

record to current.text:

or

actions:

title (aliases: getTitle)

Get the document title.

actions:

record to current.title:

value (aliases: getValue)

Get the Javascript value field of the first element node matching the selector (sel).

actions:

record to current.value:

or

actions:

wait (aliases: sleep)

Wait for the specified time.

actions:

or

waitReady

Wait until the element matching the selector (sel) is ready.

actions:

or

actions:

waitVisible

Wait until the element matching the selector (sel) is visible.

actions:

or

actions:

SSH Runner: execute commands on a remote server connected via SSH

Use ssh:// scheme to specify SSH Runner.

When step is invoked, it executes commands on a remote server connected via SSH.

runners: sc: ssh://username@hostname:port steps:

desc: 'execute `hostname`' # description of step
sc:
  command: hostname

runners: sc: hostname: hostname user: username port: 22 # host: myserver # sshConfig: path/to/ssh_config # keepSession: false # localForward: '33306:127.0.0.1:3306' # keyboardInteractive: # - match: Username # answer: k1low # - match: OTP # answer: ${MY_OTP}

See testdata/book/sshd.yml.

Structure of recorded responses

The response to the run command is always stdout and stderr.

step key or current or previous: stdout: 'hello world' # current.stdout stderr: '' # current.stderr

Exec Runner: execute command

NoteExec runner requires run:exec scope to run.

The exec runner is a built-in runner, so there is no need to specify it in the runners: section.

It execute command using command:, stdin:, shell:, background: and liveOutput:.

background: set to true to run the command in the background.

liveOutput: set to true to output the command output live.

See testdata/book/exec.yml.

Structure of recorded responses

The response to the run command is always stdout, stderr and exit_code.

step key or current or previous: stdout: 'hello world' # current.stdout stderr: '' # current.stderr exit_code: 0 # current.exit_code

Test Runner: test using recorded values

The test runner is a built-in runner, so there is no need to specify it in the runners: section.

It evaluates the conditional expression using the recorded values.

The test runner can run in the same steps as the other runners.

Dump Runner: dump recorded values

The dump runner is a built-in runner, so there is no need to specify it in the runners: section.

It dumps the specified recorded values.

or

The dump runner can run in the same steps as the other runners.

Include Runner: include other runbook

The include runner is a built-in runner, so there is no need to specify it in the runners: section.

Include runner reads and runs the runbook in the specified path.

Recorded values are nested.

It is also possible to override vars: of included runbook.

It is also possible to skip all test: sections in the included runbook.

It is also possible to force all steps in the included runbook to run.

Bind Runner: bind variables

The bind runner is a built-in runner, so there is no need to specify it in the runners: section.

It bind runner binds any values with another key.

The bind runner can run in the same steps as the other runners.

Runner Runner: Define runner in the middle of steps.

The runner runner is a built-in runner, so there is no need to specify it in the runners: section.

It defines a runner in the middle of steps.

The runner runner can not run in the same steps as the other runners.

Expression evaluation engine

runn has embedded expr-lang/expr as the evaluation engine for the expression.

See Language Definition.

Additional built-in functions

Option

See https://pkg.go.dev/github.com/k1LoW/runn#Option

Example: Run as a test helper ( func T )

https://pkg.go.dev/github.com/k1LoW/runn#T

o, err := runn.Load("testdata/**/*.yml", runn.T(t)) if err != nil { t.Fatal(err) } if err := o.RunN(ctx); err != nil { t.Fatal(err) }

Example: Add custom function ( func Func )

https://pkg.go.dev/github.com/k1LoW/runn#Func

desc: Test using GitHub runners: req: endpoint: https://github.com steps:

req:
  /search?l={{ urlencode('C++') }}&q=runn&type=Repositories:
    get:
      body:
        application/json:
          null
test: 'steps[0].res.status == 200'

o, err := runn.Load("testdata/**/*.yml", runn.Func("urlencode", url.QueryEscape)) if err != nil { t.Fatal(err) } if err := o.RunN(ctx); err != nil { t.Fatal(err) }

Scope

runn requires explicit specification of scope for some features.

runn has the following scopes.

Scope Description Default
read:parent Required for reading files above the working directory. false
read:remote Required for reading remote files. false
run:exec Required for running Exec runner. false

To specify scopes, using the --scopes option or the environment variable RUNN_SCOPES.

$ runn run path/to/**/*.yml --scopes read:parent,read:remote

$ env RUNN_SCOPES=read:parent,read:remote runn run path/to/**/*.yml

Also, runn.Scopes can be used in the code

o, err := runn.Load("path/to/**/*.yml", runn.Scopes(runn.ScopeAllowReadParent, runn.ScopeAllowReadRemote)) if err != nil { t.Fatal(err) } if err := o.RunN(ctx); err != nil { t.Fatal(err) }

To disable scope, can use !read:* instead of read:*

$ runn run path/to/**/*.yml --scopes '!read:parent'

Filter runbooks to be executed by the environment variable RUNN_RUN

Run only runbooks matching the filename "login".

$ env RUNN_RUN=login go test ./... -run TestRouter

Measure elapsed time as profile

opts := []runn.Option{ runn.T(t), runn.Book("testdata/books/login.yml"), runn.Profile(true), } o, err := runn.New(opts...) if err != nil { t.Fatal(err) } if err := o.Run(ctx); err != nil { t.Fatal(err) } f, err := os.Open("profile.json") if err != nil { t.Fatal(err) } if err := o.DumpProfile(f); err != nil { t.Fatal(err) }

or

$ runn run testdata/books/login.yml --profile

The runbook run profile can be read with runn rprof command.

$ runn rprof runn.prof runbooklogin site 2995.72ms steps[0].req 747.67ms steps[1].req 185.69ms steps[2].req 192.65ms steps[3].req 188.23ms steps[4].req 569.53ms steps[5].req 299.88ms steps[6].test 0.14ms steps[7].include 620.88ms runbookinclude 605.56ms steps[0].req 605.54ms steps[8].req 190.92ms [total] 2995.84ms

Capture runbook runs

opts := []runn.Option{ runn.T(t), runn.Capture(capture.Runbook("path/to/dir")), } o, err := runn.Load("testdata/books/**/*.yml", opts...) if err != nil { t.Fatal(err) } if err := o.RunN(ctx); err != nil { t.Fatal(err) }

or

$ runn run path/to/**/*.yml --capture path/to/dir

Load test using runbooks

You can use the runn loadt command for load testing using runbooks.

$ runn loadt --load-concurrent 2 --max-rps 0 path/to/*.yml

Number of runbooks per RunN....: 15 Warm up time (--warm-up).......: 5s Duration (--duration)..........: 10s Concurrent (--load-concurrent).: 2 Max RunN per second (--max-rps): 0

Total..........................: 12 Succeeded......................: 12 Failed.........................: 0 Error rate.....................: 0% RunN per seconds...............: 1.2 Latency .......................: max=1,835.1ms min=1,451.3ms avg=1,627.8ms med=1,619.8ms p(90)=1,741.5ms p(99)=1,788.4ms

It also checks the results of the load test with the --threshold option. If the condition is not met, it returns exit status 1.

$ runn loadt --load-concurrent 2 --max-rps 0 --threshold 'error_rate < 10' path/to/*.yml

Number of runbooks per RunN...: 15 Warm up time (--warm-up)......: 5s Duration (--duration).........: 10s Concurrent (--load-concurrent): 2

Total.........................: 13 Succeeded.....................: 12 Failed........................: 1 Error rate....................: 7.6% RunN per seconds..............: 1.3 Latency ......................: max=1,790.2ms min=95.0ms avg=1,541.4ms med=1,640.4ms p(90)=1,749.7ms p(99)=1,786.5ms

Error: (error_rate < 10) is not true error_rate < 10 ├── error_rate => 14.285714285714285 └── 10 => 10

Variables for threshold

Variable name Type Description
total int Total
succeeded int Succeeded
failed int Failed
error_rate float Error rate
rps float RunN per seconds
max float Latency max (ms)
mid float Latency mid (ms)
min float Latency min (ms)
p90 float Latency p(90) (ms)
p99 float Latency p(99) (ms)
avg float Latency avg (ms)

Install

As a CLI tool

deb:

$ export RUNN_VERSION=X.X.X $ curl -o runn.deb -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.deb $ dpkg -i runn.deb

RPM:

$ export RUNN_VERSION=X.X.X $ yum install https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.rpm

apk:

$ export RUNN_VERSION=X.X.X $ curl -o runn.apk -L https://github.com/k1LoW/runn/releases/download/v$RUNN_VERSION/runn_$RUNN_VERSION-1_amd64.apk $ apk add runn.apk

homebrew tap:

$ brew install k1LoW/tap/runn

aqua:

manually:

Download binary from releases page

docker:

$ docker container run -it --rm --name runn -v $PWD:/books ghcr.io/k1low/runn:latest list /books/*.yml

go install:

$ go install github.com/k1LoW/runn/cmd/runn@latest

As a test helper

$ go get github.com/k1LoW/runn

Alternatives

References

License