Semaphore Toolbox | Semaphore (original) (raw)

The toolbox is an open source package of command line tools available in every Semaphore job. It provides utility functions to clone, cache data, run test services, or change language versions. This page explains all the available tools.

Overview​

The Semaphore toolbox is preinstalled in all Semaphore jobs. It is also installed in self-hosted agents, so you can assume that all the tools described on this page are available for use.

artifact​

This is the command line tool to access the Semaphore artifacts.

The usage is:

artifact <command> <namespace> <file_or_directory> [flags]

The available commands are:

The available namespaces are:

See artifact namespaces for more details.

The optional flags are:

Examples​

Artifact usage examples

# force push a final deliverable to the project level
artifact push project app-v1.tar.gz --force

# push a screenshot to the job level
artifact push job screenshots/app.png

# push a binary to the workflow level 
artifact push workflow build/app

# pull a binary from the workflow level 
artifact pull workflow build/app

# delete a binary from the workflow level 
artifact yank workflow build/app

Error status​

The artifact pull and artifact yank commands exit with non-zero status (ending the job with error) when:

Supported filenames​

The uploaded files must meet the following requirements:

You can workaround these limitations by compressing the file with tar before pushing it to the artifact store. For example:

Creating a tarball before storing the artifact

tar -czvf example.tar.gz ~/example
artifact push workflow example.tar.gz

To retrieve the files, use:

Retrieving the tarball

artifact pull workflow example.tar.gz
tar -xzf example.tar.gz

cache​

The cache tool lets you interact with your project's Semaphore cache.

The syntax is:

cache <command> <keys> [path] [flags]

The available commands are:

Cache keys​

The cache keys are a comma-separated list of strings to label or select cached items.

The following examples show different ways to store and identify Ruby Gems in the vendor/bundle directory:

cache store our-gems vendor/bundle
cache store gems-$SEMAPHORE_GIT_BRANCH vendor/bundle
cache store gems-$SEMAPHORE_GIT_BRANCH-revision-$(checksum Gemfile.lock) vendor/bundle

To restore counterparts for the example above are:

cache restore our-gems
cache restore gems-$SEMAPHORE_GIT_BRANCH
cache restore gems-$SEMAPHORE_GIT_BRANCH-revision-$(checksum Gemfile.lock),gems-master

cache restore always retrieves the first item that matches a key found in the cache. The rest are ignored. If no archives are restored the command exits with exit status 0.

note

As the cache store uses tar, which automatically removes any leading / from a given path value, any further changes of the path after the store command completes will not be automatically propagated to the cache.

Space management​

The cache store command automatically deletes older files when the cache is full. You can change the deletion criteria using the --cleanup-by or -c argument. For example:

cache store our-gems vendor/bundle --cleanup-by SIZE

The supported options for --cleanup-by are:

Environment variables​

The cache tool depends on the following environment variables:

checkout​

This tool clones the repository using Git. For performance reasons, the default behavior is to perform a shallow clone. Shallow clones only include the latest commit instead of the full repository history.

The checkout command takes no arguments. It also changes the directory into the cloned repository. So, for example, this would work on a Node.js project:

No need to cd into the cloned repository

Full clone​

If you want to do a full clone of the repository you can run the following commands to "unshallow" a repository:

checkout
git fetch --unshallow
git config remote.origin.fetch '+refs/heads/*:refs/remotes/origin/*'
git fetch --all

Clone with cache​

info

This is marked as an experimental feature.

If a full clone takes too long, you can use checkout --use-cache to save a copy of the repository in the Semaphore cache. This should speed up full clones significantly.

To perform a full repository clone with cache run:

This option creates or refreshes one or more copies of the repository in the Semaphore cache. See the environment variables to learn how to control the behavior of the cached copies.

Environment variables​

The checkout command uses the following environment variables.

checksum​

This tool takes a single argument which is the file to checksum. It outputs the MD5 checksum of the file's contents. This tool is useful for tagging artifacts or generating cache keys.

The syntax is:

Example:

Checksumming package-lock.json

$ checksum package-lock.json
3dc6f33834092c93d26b71f9a35e4bb3

install-package​

The install-package tool is used to manage Ubuntu packages you may need for your jobs. It downloads and caches packages in a way that can be quickly reinstalled over and over again in different jobs. This is a tool made for convenience, but you can still use sudo to install packages using the system's package manager.

The syntax is:

install-package syntax

install-package <flag> <pkg-name>

Where <pkg-name> is the Ubuntu package name without the .deb extension. In other words, you want libc6 instead of libc6.deb.

Where the flag is one of the following:

You can supply multiple packages with their versions in the same invocation:

 install-package install mongodb-clients=3.6.8 mysql-client=8.0.36-0ubuntu0.20.04.1

The tool integrates with the Semaphore cache to save, retrieve, and update the Deb packages as needed.

You can reinstall the packages in a different job within the same project with:

Installing packages from the cache in other jobs

retry​

The retry tool can be used to retry a command on an interval. This is useful when waiting for resources to become available or to mitigate connectivity issues.

The syntax is:

Where flags are optional arguments:

For example, to retry 5 times bundle install with a 10-second sleep use:

retry --times 5 --sleep 10 bundle install

sem-context​

This tool enables sharing key-value data between Semaphore jobs within the same workflow. It carries values from parent jobs, and updates made within a workflow are inherited by promoted jobs. However, please note that values are not shared across different promotions.

note

sem-context depends on artifacts to store data.

The syntax is:

sem-context syntax

sem-context <action> <key>[=value] [flags]

The possible actions are:

The following limitations apply to keys and values:

Possible flags are:

put key​

The following example stores the version number with the key ReleaseVersion.

sem-context put ReleaseVersion=1.2.3

Exit status codes:

get key​

The following example retrieves the version number using the key ReleaseVersion.

$ sem-context get ReleaseVersion
1.2.3

Exit status codes:

delete key​

The following example deletes the version value using the key ReleaseVersion.

sem-context delete ReleaseVersion

Exit status codes:

sem-service​

The sem-service tool manages databases and other useful services in Ubuntu-based environments.

The syntax is:

sem-service syntax

sem-service <command> <service> [version] [flags]

Where <command> is one of the following:

All services run locally on the agent running the job. Started services always listen on 0.0.0.0 and their default port.

Services supported​

The <service> argument is one of the following:

The <version> argument depends on the kind of service being started. You can run sem-service help to see all available services and versions.

Options for MySQL and PostgreSQL​

When starting mysql or postgres services you can provide the following optional arguments:

Container registry images​

The sem-service tool pulls images from the Semaphore Container Registry.

sem-version​

The sem-version tool manages language and utility versions in Ubuntu environments. It provides a quick and simple way to install and switch to a version of a command line tool or programming language.

The syntax is:

sem-version syntax

sem-version <target> <version> [flags]

Languages supported​

Where <target> is one of the following:

The <version> depends on the target used. The command fails with a non-zero exit code unless the --ignore or -i flag is supplied.

For example, to download and use Go version 1.22:

spc​

The Semaphore Pipeline Compiler (SPC) is used during initialization jobs to process runtime values in input pipelines. It generates an output pipeline that is used for the workflow execution.

note

spc is called automatically by Semaphore during initialization.

The syntax is:

spc syntax

spc compile --input <input-pipeline-file> --output <output-pipeline-file> --logs <logs-file>

The tool evaluates change_in arguments and outputs the result of each match condition.

test-results​

This tool processes test results in the JUnit format. It is used to process and generate test reports and the flaky tests dashboard.

The syntax is:

test-results <command> [flags]

The available commands are:

The available flags are:

The test-results CLI is open-sourced and available on semaphoreci/test-results. You can access the source code, contribute, and report issues there.

Merging test results​

To use the test result feature you must add the following command at the end of every test job.

The syntax is:

test-results publish <path>

Where <path> is a file or a directory containing JUnit XML files. This processes the files and uploads the results to the artifact store.

If working with Docker containers, you need to expose the JUnit files using bind mounts. For example:

docker run -v $(pwd):/app test-runner run-specs
test-results publish junit.xml

sem-semantic-release​

sem-semantic-release wraps the [semantic release CLI] and exports the release information to the rest of your delivery process using sem-context put. This information can later be retrieved with sem-context get.

It is handy when you want to continue the delivery process once the release has been created. A common example would be to build a Docker image tagged with the release version, or add an annotation to a Kubernetes deployment manifest.

General usage in CI:

checkout
sem-semantic-release

Usage options:

Usage: sem-semantic-release [OPTION]...

Options:
  --dry-run   runs semantic-release without publishing version
  --plugins   npm plugins and extensions to be installed
  --branches  branches to run semantic release for
  --version   semantic-release version

The sem-semantic-release exports the following values to sem-context

Key Description
ReleasePublished Whether a new release was published ("true" or "false")
ReleaseVersion Version of the new release. (e.g. "1.3.0")
ReleaseMajorVersion Major version of the new release. (e.g. "1")
ReleaseMinorVersion Minor version of the new release. (e.g. "3")
ReleasePatchVersion Patch version of the new release. (e.g. "0")

You can use these values in your pipeline to automatically figure version numbers. For example:

docker build . -t my-app:$(sem-context get ReleaseVersion)
docker push
sed -i "s/IMAGE/$(sem-context get ReleaseVersion)" deployment.yml
kubectl apply -f deployment.yml

See also​