Run Splunk Enterprise as a systemd service (original) (raw)

This documentation does not apply to the most recent version of Splunk® Enterprise. For documentation on the most recent version, go to the latest release.

Splunk Enterprise 7.2.2 and higher provides support for systemd on Linux with an enhanced enable boot-start command that lets you automatically configure systemd to manage splunkd as a service.

What is systemd?

systemd is a system startup and service manager that is widely deployed as the default init system on most major Linux distributions. You can configure systemd to manage processes, such as splunkd, as services, and allocate system resources to those processes under cgroups.

systemd advantages

systemd offers the following general advantages:

systemd offers these additional specific advantages for Splunk deployments:

Configure systemd to manage splunkd

You can use either of the following two methods to configure systemd to manage splunkd as a service:

If you configure systemd using enable boot-start, a Splunk service unit file is created automatically. No additional manual configuration is required.

System requirements

Workload management supports Linux cgroups v1 only. If your Linux system has been upgraded to a version running cgroups v2, you must revert your system to cgroups v1 to use Workload Management in Splunk Enterprise. For additional guidance and resources, see Workload Management operation during the transition to cgroups v2.

Permissions requirements

The enable boot-start command and systemd have the following permissions requirements:

For instructions on how to create a new user with super user permissions, see your Linux documentation.

Unprivileged users must use sudo to run splunk start|stop|restart. If you do not use sudo and attempt to run splunk start|stop|restart when managed by systemd, a prompt appears requesting authentication. For example:

==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units === Authentication is required to manage system services or units. Multiple identities can be used for authentication:

  2. Choose identity to authenticate as (1-2): 2 Password: ==== AUTHENTICATION COMPLETE ===

Alternately, you can install polkit rules with the enable boot-start command to allow unprivileged users to run start|stop|restart operations under systemd without using sudo. For instructions, see Install polkit rules to elevate user permissions.

Unit file naming considerations

The enable boot-start command creates a systemd unit file named Splunkd.service. The unit file name is based on the SPLUNK_SERVER_NAME in splunk-launch.conf, which is set by default to Splunkd.

If for any reason you remove the SPLUNK_SERVER_NAME value from splunk-launch.conf, enable boot-start creates a unit file named splunkd.service (lower case "splunkd") and sets SPLUNK_SERVER_NAME=splunkd in the splunk-launch.conf file.

You can specify a different name of your choice for the unit file when you run enable boot-start. See Specify a different unit file name.

Configure systemd using enable boot-start

You can configure systemd to manage splunkd as a service using the enable boot-start command, as follows:

  1. Log into the machine on which you want to configure systemd to manage splunkd as a service.
  2. Stop splunkd.
    $SPLUNK_HOME/bin/splunk stop
  3. If you previously enabled Splunk Enterprise to start at boot using the enable boot-start command, run disable boot-start to remove the splunk init script located in /etc/init.d and its symbolic links.
    [sudo] $SPLUNK_HOME/bin/splunk disable boot-start
    For instructions on how to reinstall the splunk init script, see Install splunk init script.
  4. Run the enable boot-start command, specifying the -systemd-managed, -user, and -group parameters, as follows:
    [sudo] $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user -group
    Specifying -user and -group is optional but recommended. If you do not specify -user, the SPLUNK_OS_USER in splunk-launch.conf is used. If SPLUNK_OS_USER is not defined, the owner of the splunk binary is used.
    This installs the following systemd service unit file, named Splunkd.service by default, in /etc/systemd/system. To specify a different unit file name, use the -systemd-unit-file-name option. See Specify a different unit file name.

#This unit file replaces the traditional start-up script for systemd
#configurations, and is used when enabling boot-start for Splunk on
#systemd-based Linux distributions.
[Unit]
Description=Systemd service file for Splunk, generated by 'splunk enable boot-start'
After=network.target
[Service]
Type=simple
Restart=always
ExecStart=/opt/splunk/bin/splunk _internal_launch_under_systemd
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=360
LimitNOFILE=65536
SuccessExitStatus=51 52
RestartPreventExitStatus=51
RestartForceExitStatus=52
User=splunk
Group=splunk
Delegate=true
CPUShares=1024
MemoryLimit=
PermissionsStartOnly=true
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/cpu/system.slice/%n"
ExecStartPost=/bin/bash -c "chown -R splunk:splunk /sys/fs/cgroup/memory/system.slice/%n"
[Install]
WantedBy=multi-user.target
The MemoryLimit value is set to the total system memory available in bytes when the service unit file is created. The MemoryLimit value will not update if the total available system memory changes. To update the MemoryLimit value in the unit file, you can manually edit the value or use the boot-start command to disable and re-enable systemd.
The following unit file properties are required. Do not change these values without appropriate guidance.
Type=simple
Restart=always
ExecStart=$SPLUNK_HOME/bin/splunk _internal_launch_under_systemd
Delegate=true This property is required for workload management. See Configure workload management.
Do not use the following properties. These properties can cause splunkd to fail on restart.
RemainAfterExit=yes
ExecStop
For more information, see Systemd unit file properties. 5. Start splunkd.
[sudo] $SPLUNK_HOME/bin/splunk start
This starts splunkd as a systemd service.
Under systemd, splunk start|stop|restart commands are mapped to systemctl start|stop|restart commands. 6. Verify that splunkd is running as a systemd service. For example:
$SPLUNK_HOME/bin/splunk status
splunkd is running (PID: 24772).
splunk helpers are running (PIDs: 24843 24857 24984 25032).
Alternatively, you can use systemctl status to check if the splunkd process is running. However, when using this command, a brief time lag can occur during which systemctl status shows "active" and splunk status shows "splunkd is not running".
Configuring systemd to manage splunkd as a service creates CPU and Memory cgroups in these locations:
CPU: /sys/fs/cgroup/cpu/system.slice/Splunkd.service
Memory: /sys/fs/cgroup/memory/system.slice/Splunkd.service 7. For distributed deployments, repeat steps 1-6 on all search heads and indexers.

Additional options for enable boot-start

The enable boot-start command supports these additional options:

Install splunk init script

In version 7.2.2 and higher, the enable boot-start command adds a -systemd-managed 0|1 option that controls whether to install the splunk init script in /etc/init.d or the Splunkd.service unit file in /etc/systemd/system.

To install the splunk init script, specify -systemd-managed 0:

$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 0 -user Init script installed at /etc/init.d/splunk. Init script is configured to run at boot.

See Configure Splunk Enterprise to start at boot time.

In version 7.2.2 through 7.2.x, if you do not specify the -systemd-managed option, the enable boot-start command defaults to -systemd-managed 1 and installs the Splunkd.service unit file. In version 7.3.0 and later, this default behavior is reversed, and the enable boot-start command defaults to -systemd-managed 0 and installs the splunk init file.

The init.d boot-start script is not compatible with RHEL 8 and higher.

Specify a different unit file name

The default splunkd unit file name is Splunkd.service. You can specify a different name for the unit file and update the SPLUNK_SERVER_NAME value in splunk-launch.conf using the -systemd-unit-file-name option. For example, to create a unit file with the name "splunk.service":

$SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -systemd-unit-file-name splunk
Systemd unit file installed at /etc/systemd/system/splunk.service. Configured as systemd managed service.

For more information, see Unit file naming considerations.

Install polkit rules to elevate user permissions

In version 8.1.1 and higher, the enable boot-start command adds an option to install polkit rules that allow non-root users to run start, stop, and restart operations under systemd without using sudo. Installing the polkit rules can reduce overhead for admins that must otherwise add unprivileged users to the sudoers file to run these operations under systemd.

To install polkit rules:

Run the enable boot-start command, specifying the -create-polkit-rules option, as follows:

./splunk enable boot-start -systemd-managed 1 -create-polkit-rules 1 -user

If you previously ran enable boot-start and specified a different user, you must change the owner of $SPLUNK_HOME to the new user for whom you create the polkit rules. For example:

chown -R $SPLUNK_HOME

Before you can install polkit rules using the create-polkit-rules option, you must install the Polkit library on your system if you have not already done so.

Configure systemd on a clean install

To configure systemd on a clean installation of Splunk Enterprise:

  1. Expand the install package in an appropriate directory. For example:
    tar xvfz splunk_package_name.tgz -C /opt
  2. Run enable boot-start to install the Splunkd.service unit file:
    sudo $SPLUNK_HOME/bin/splunk enable boot-start -systemd-managed 1 -user
    When running enable boot-start for the first time after a clean install, Splunk Enterprise prompts you to accept the Splunk software license agreement. To automatically accept the license without prompt, specify the --accept-license flag with the command.
  3. Start splunkd.
    sudo $SPLUNK_HOME/bin/splunk start
  4. Verify that splunkd is running as a systemd service.
    $SPLUNK_HOME/bin/splunk status

Manage clusters under systemd

When managing an indexer cluster under systemd:

When managing a search head cluster under systemd:

Upgrade considerations for systemd

Upgrade from 8.0.x to 8.1

If you configured Splunk Enterprise version 8.0.x to run as a systemd service, upon upgrade to version 8.1, Splunk Enterprise adds the following properties to the Splunkd.service unit file:

When Splunk Enterprise adds these unit file properties, it creates a new unit file that replaces the existing unit file Splunkd.service. It also renames the old unit file Splunkd.service_<timestamp>, which it saves for backup purposes only.

When upgrading directly from 7.3.x or lower to 8.1, Splunk Enterprise adds the Group property to the unit file.

Upgrade from 7.3.x or lower to 8.0

If you configured Splunk Enterprise version 7.3.x or lower to run as a systemd service, upon upgrade to version 8.0.x, on initial start, Splunk Enterprise modifies the existing systemd configuration as follows:

You must use sudo splunk start to perform the initial start of Splunk Enterprise after installing the version 8.0.0 upgrade tarball.

Using systemctl start to perform the initial start of Splunk Enterprise on upgrade to version 8.0.0 will fail.

For detailed information on upgrading Splunk Enterprise, see How to upgrade Splunk Enterprise in the Installation Manual.