Jenkins : Swarm Plugin (original) (raw)

This plugin enables slaves to auto-discover nearby Jenkins master and join it automatically, thereby forming an ad-hoc cluster.
This plugin consists of two pieces:

1. A self-contained CLI client that discovers a nearby Jenkins (via a UDP broadcast) and joins it
2. A plugin that needs to be installed on Jenkins master to accept swarm clients

The primary use case of this came from the idea of a "Jenkins slave appliance" — a virtual machine pre-packaged with the swarm client CLI — that someone can just grab and run.

With the swarm client, a person who's willing to contribute some of his computing power to the cluster just needs to run this virtual machine, and the cluster automatically gets additional resources. Because it's packaged as a VM, there is no need to worry about the builds/tests interfering with the host system or altering its settings unexpectedly.

Usage

1. Install the swarm plugin from the Update Center
2. Download the CLI agent from here.
3. Run the CLI agent with java -jar path/to/swarm-client.jar. There are no required command-line options; run with the -help option to see the available options.

Documentation

• Proxy Configuration
• Logging

Available Options

$ java -jar swarm-client.jar --help -autoDiscoveryAddress VAL : Use this address for UDP-based auto-discovery (default 255.255.255.255) -candidateTag VAL : Show swarm candidate with tag only -deleteExistingClients : Deletes any existing slave with the same name. -description VAL : Description to be put on the slave -disableClientsUniqueId : Disables client's unique ID. -disableSslVerification : Disables SSL verification in the HttpClient. -executors N : Number of executors -fsroot FILE : Directory where Jenkins places files -help (--help) : Show the help screen -labels VAL : Whitespace-separated list of labels to be assigned for this slave. Multiple options are allowed. -labelsFile VAL : File location with space delimited list of labels. If the file changes, the client is restarted. -master VAL : The complete target Jenkins URL like 'http://server:8080/jenkins/'. If this option is specified, auto-discovery will be skipped -maxRetryInterval N : Max time to wait before retry in seconds. Default is 60 seconds. -mode MODE : The mode controlling how Jenkins allocates jobs to slaves. Can be either 'normal' (utilize this slave as much as possible) or 'exclusive' (leave this machine for tied jobs only). Default is 'normal'. -name VAL : Name of the slave -noRetryAfterConnected : Do not retry if a successful connection gets closed. -password VAL : The Jenkins user password -passwordEnvVariable VAL : Environment variable that the password is stored in -passwordFile VAL : File containing the Jenkins user password -pidFile VAL : File to write PID to -retry N : Number of retries before giving up. Unlimited if not specified. -retryBackOffStrategy RETRY_BACK_OFF_S : The mode controlling retry wait time. TRATEGY Can be either 'none' (use same interval between retries) or 'linear' (increase wait time before each retry up to maxRetryInterval) or 'exponential' (double wait interval on each retry up to maxRetryInterval). Default is 'none'. -retryInterval N : Time to wait before retry in seconds. Default is 10 seconds. -showHostName (--showHostName) : Show hostname instead of IP address -sslFingerprints VAL : Whitespace-separated list of accepted certificate fingerprints (SHA-256/Hex), otherwise system truststore will be used. No revocation, expiration or not yet valid check will be performed for custom fingerprints! Multiple options are allowed. -t (--toolLocation) : A tool location to be defined on this slave. It is specified as 'toolName=location' -tunnel VAL : Connect to the specified host and port, instead of connecting directly to Jenkins. Useful when connection to Jenkins needs to be tunneled. Can be also HOST: or :PORT, in which case the missing portion will be auto-configured like the default behavior -username VAL : The Jenkins username for authentication

FAQ

Error configuration : "xxx.xxx.xxx.xxx doesn't have the URL configuration yet"

This error indicates that the Jenkins master running at the given URL has never configured, and therefore the swarm client cannot determine its HTTP URL. To fix this problem, please go to the system configuration screen of this Jenkins, and click "submit".

Changelog

Version 3.17 (June 2, 2019)

• Swarm Plugin
  • Add integration test framework (#81)
  • Add test for JENKINS-39443 (#100)
  • Miscellaneous code cleanup (#86, #91, #92, #93, #94, #97, #101, #108, #109)
• Swarm Client
  • Fix SECURITY-1252 XML External Entity (XXE) vulnerability (#84, #90, #96)
  • Rework logging of command-line arguments (#95)
  • Fix JENKINS-42930 Use System proxy when available and migrate from Commons HttpClient 3.x to HttpComponents Client 4.x (#105, Documentation)
  • Fix JENKINS-45295 Swarm Client should update labels on the fly when labels file changes (#104, #110)
  • Fix JENKINS-50970 SLF4J logging not working in Swarm client (#98)
  • Clean up logging subsystem (#99, #102, #106, Documentation)
  • Fix invalid exit code (#103)
  • Remove deprecated -logFile parameter (#107)

Version 3.16 (May 21, 2019)

• Swarm Plugin
  • Update minimum Jenkins core requirement to 2.60.3 (#87)
  • Remove unnecessary field (#85)
• Swarm Plugin Client
  • Disable DTDs completely in all XML parsers to prevent XML External Entity (XXE) attacks (#84)
  • Update Jenkins Remoting version from 3.28 to 3.30 (#78, full changelog)

Version 3.15 (2018 Dec 12)

• Swarm Plugin
  • PR #75 - Fix the label removal in the "removeLabels" endpoint
• Swarm Plugin Client
  • Update Jenkins Remoting version from 3.26 to 3.28 to pick up new stability fixes (full changelog)

Version 3.14 (2018 Sep 04)

• Swarm Plugin Client
  • Update Remoting from 3.21 to 3.26 in order to pick new stability fixes (full changelog)
• Swarm Plugin
  • Update Swarm Plugin Client to 3.26

Version 3.13 (2018 Jun 08)

• Swarm Plugin
  • Update minimal Jenkins Core requirement to 2.60.1
• Swarm Plugin Client
  • Update Remoting to 3.21 to pick logging and "no_proxy" handling fixes
    * Full changelog

Version 3.12 (2018 Mar 22)

• Swarm Plugin
  • No changes
• Swarm Plugin Client
  • JENKINS-50237 - Update Remoting from 3.18 to 3.19 to pick up the exception propagation fix
    * Full changelog

Version 3.11 (2018 Mar 19)

• Swarm Plugin
  • No changes
• Swarm Plugin Client
  • JENKINS-50252 - Update Remoting from 3.16 to 3.18 to pick up bug fixes and serialization diagnosability improvements
    * Remoting Changelog
  • PR #68 - The plugin now trims input strings for password files specified in "-passwordFile"

Version 3.10 (2018 Feb 21)

• Swarm Plugin
  • PR #62 - Add ability to download the client directory from the plugin installed in Jenkins
    * URL: ${JENKINS_URL}/swarm/swarm-client.jar
• Swarm Plugin Client
  • No changes

Version 3.9 (2018 Feb 07)

• Swarm Plugin
  • JENKINS-49292 - Reduce log level from ALL to INFO in sample logging.properties to reduce log spam
• Swarm Plugin Client
  • PR #66 - Add support of the "–passwordFile" option

Version 3.8 (2018 Jan 10)

• Swarm Plugin:
  • No changes
• Swarm Plugin Client:
  • PR #65 - Update Remoting from 3.15 to 3.16
  • • Remoting Changelog

Version 3.7 (2017 Dec 22)

• Swarm Plugin:
  • No changes
• Swarm Plugin Client:
  • PR #63 - Require Java 8 (client-side only)
  • PR #63 - Update Remoting from 3.10.2 to 3.15
  • • Remoting Changelog
  • PR #61 - Prevent the infinite reconnect cycle in Remoting Launcher, use the client's failover logic instead

Version 3.6 (2017 Oct 18)

• Update Remoting in Swarm Client from 3.4.1 to 3.10.2
  • Remoting Changelog
• PR #55 - Introduce the "-pidFile" option, which creates a file with the process PID
  • Errata: The current implementation may cause File descriptor leaks in edge cases
• JENKINS-43674 - Prevent NullPointerException in Swarm client in HTTPS mode without -disableSslVerification or -sslFingerprints
• JENKINS-42098 - Prevent LinkageError when building a Maven project on a swarm node with new Maven versions

Version 3.5 (2017 Oct 11)

• SECURITY-597 - Swarm Plugin Client bundled version of the commons-httpclient library, which was vulnerable to MiTM

Version 3.4 (2017 Apr 10)

• Add option -sslFingerprints providing a possibility to add custom SSL trust anchors without adding them to the system store.

Version 3.3 (2017 Feb 10)

• Finally a release!
• Added -logFile and -labelsFile options. Now supports dynamic labels
• Added support for very large numbers of dynamic labels when using -labelsFile
• Remove consecutive slashes in plugin urls
• docker-compose configuration updates
• Add retry backoff strategy
• Bumped remoting library to same as Jenkins LTS at the moment
• Updates to make build and testing pass with new Jenkins plugin parent pom work

Version 3.2 (2017 Feb 8)

• Failed to release due to INFRA-588

Version 3.1 (2017 Feb 8)

• Failed to release due to INFRA-588

Version 3.0 (2016 Dec 27)

• Failed to release due to INFRA-588

Version 2.3 (2016 Nov 28)

• Failed to release due to INFRA-588

Version 2.2 (2016 July 26)

• Failed to release due to INFRA-588

Version 2.1 (2016 May 20)

• Implement #28917 - Update remoting to one supported by latest LTS
• MESOS_TASK_ID used as Jenkins slave ID if available as environment variable (for Mesos/Marathon integration).
• Updating jenkins remoting dependency. Swarm client now matches the jenkins-remoting version in Jenkins-1.625.3 LTS
• Implement #34593 - add an option to delete existing clients
• Add integration test environment based upon Docker compose

Version 2.0 (2015 Aug 3)

• Implement #28148 - Whitespaces in toolLocations, (breaking change, see pull request 28)
• Add ability to disable unique ID for Clients. (see pull request 33)
• Remove unused code and reformat source files.

Version 1.26 (2015 Jul 21)

• Re-release of 1.25, some artefacts was not properly deployed.

Version 1.25 (2015 Jul 21)

• Correct #29232 - Set the HTTP Connection:close header to ensure the underlying socket is closed (see pull request 29)
• Add a markdown formatted README to better describe the project for Github viewers
• Improve end user reporting of hostname lookup errors (see pull request 30)
• Make javadoc compile with JDK 8

Version 1.24 (2015 Apr 28)

• Correct #26558 - Clients should provide a unique ID to be used for name collision avoidance (see pull request 26)
• Improve printout when Jenkins master is not configured with a URL (see pull request 27)

Version 1.23 (2015 Apr 27)

• Add the tunnel option to pass it to the Jenkins engine (see pull request 22)
• Minor enhancements to make the swarm client usable for mere detection of Jenkins instances. (see pull request 23)
• Correct #24149 - LogConfigurationException (see pull request 24)
• Computer.toNode() can return null (see pull request 25)

Version 1.22 (2014 Nov 28)

• added new option 'passwordEnvVariable' (see pull request 21)

Version 1.21 (2014 Nov 6)

• Instead of constructing the tool location key, just use the existing descriptor (#25064, see pull request 20)
• Use latest Jenkins LTS remoting library (1.580.1 Jenkins LTS version)

Version 1.20 (2014 Oct 8)

• Fixup on handling of tool locations on Windows (#25002, see pull request 19)

Version 1.19 (2014 Oct 6)

• Correct bug introduced by 1.18 where the client did not work _unless_ you set tool locations (#24995, see pull request 18)

Version 1.18 (2014 Oct 2)

• Set tool locations from Swarm plugin CLI, (#7543, see pull request 17)

Version 1.17 (2014 Sep 30)

• Add -noRetryAfterConnected and -retry options. These provide optional exit strategies for the default unlimited retry loop.
• Require a well-formed master URL, ensuring trailing slash
• [FIXED JENKINS-21892] Update swarm client to send CSRF token
• Use latest releases of commons-codec, args4j, and remoting

Version 1.16 (2014 Jul 1)

• Bump remoting to match Jenkins LTS (issue #22730, see pull request 14)

Version 1.15

• undocumented, or maybe a typo of 1.12?

Version 1.12, 1.11 (2014 Jan 15)

• Use compatible version of commons codec (JENKINS-21155, see pull request 7 and pull request 8).

Version 1.10 (2013 Oct 21)

• swarm 1.9 can't connect to current LTS as slave.jar too old (JENKINS-20138).

Version 1.9 (2013 May 18))

• Add option for specifying Node.Mode(pull #3).

Version 1.8 (2012 Nov 21)

• Changing broadcast to send a UDP packet payload of 128 bytes instead of 0
• Allow slave connection without requiring UDP
• Adds disableSslVerification option

Version 1.6 (2012 Mar 18)

• fixed references from Hudson to Jenkins.
• Swarm Client fails to connect to Jenkins when Authentication is enabled but Authorization is diabled (JENKINS-11663).
• support remoting 2.12.

Version 1.5 (2011 Aug 11)

• check whether user has SlaveComputer.CREATE permission.
• allow authentication in swarm plugin (JENKINS-5504).

Version 1.4 (2010 Aug 14)

• Fixed broken help links.
• Node properties save correctly.
• Added Japanese localization.

Version 1.3 (2010 Jan 14)

• Fixed a packaging problem in the client jar (JENKINS-5275)

Version 1.2 (2009 Dec 30)

• Minor text correction

Version 1.1 (2009 Jul 15)

• Added the -master option.

Version 1.0 (2009 May 23)

• Initial release