Cluster endpoint descriptions - Splunk Documentation (original) (raw)
Manage indexer clusters and search head clusters in Splunk Enterprise.
To distinguish indexer cluster endpoints from search cluster endpoints, note:
- Indexer cluster endpoints: Endpoints that contain
cluster
in their URIs pertain to indexer clusters. - Search head cluster endpoints: Endpoints that contain
shcluster
in their URIs pertain to search head clusters.
The values manager
and peer
replace the prior values of master
and slave
. The prior values are currently still supported, but they will be removed from the product in a future release.
Usage details
Review ACL information for an endpoint
To check Access Control List (ACL) properties for an endpoint, append /acl
to the path. For more information see Access Control List in the REST API User Manual.
Username and password authentication is required for access to endpoints and REST operations.
Splunk users must have role and/or capability-based authorization to use REST endpoints. Users with an administrative role, such as admin
, can access authorization information in Splunk Web. To view the roles assigned to a user, select Settings > Access controls and click Users. To determine the capabilities assigned to a role, select Settings > Access controls and click Roles.
App and user context
Typically, knowledge objects, such as saved searches or event types, have an app/user context that is the namespace. For more information about specifying a namespace, see Namespace in the REST API User Manual.
Splunk Cloud Platform limitations
As a Splunk Cloud Platform user, you are restricted to interacting with the search tier only with the REST API. Cluster endpoints are generally not accessible in Splunk Cloud Platform.
Authorized users can access and configure other indexer cluster nodes, including indexer or cluster manager, or license manager nodes, in the Splunk Cloud Platform manager user interface.
See Access requirements and limitations for the Splunk Cloud Platform REST API in the the REST API Tutorials manual for more information.
Indexer cluster endpoints
The endpoints in this section pertain to indexer clusters.
All endpoints that contain cluster
in their URIs pertain to indexer clusters. In this section, if a URI contains the term search head, it refers to search head nodes in the indexer cluster. The term peer node refers to peer nodes in the indexer cluster. For more information about indexer cluster architecture, see The basics of indexer cluster architecture and Search head configuration overview in the Managing Indexers and Clusters of Indexers manual.
cluster/config
https://:/services/cluster/config
Access cluster node configuration details.
GET
List cluster node configuration.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description | ||
---|---|---|---|
cxn_timeout | Low-level timeout, in seconds, for establishing connection between cluster nodes. Defaults to 60 seconds. | ||
disabled | Indicates if this node is disabled. | ||
forwarderdata_rcv_port | The port from which to receive data from a forwarder. | ||
forwarderdata_use_ssl | Indicates whether to use SSL when receiving data from a forwarder. | ||
heartbeat_period | Only valid for peer nodes in a cluster. The time, in seconds, that a peer attempts to send a heartbeat to the manager | ||
heartbeat_timeout | Only valid for the manager node in a cluster configuration. The time, in seconds, before a manager considers a peer down. Once a peer is down, the manager initiates steps to replicate buckets from the dead peer to its live peers. Defaults to 60 seconds. | ||
manager_uri | Valid only for nodes configured as a peer or searchhead.URI of the cluster manager to which this node connects. | ||
max_peer_build_load | The number of jobs that a peer can have in progress at any time that make the bucket searchable. | ||
max_peer_rep_load | Maximum number of replications that can be ongoing as a target. | ||
mode | Valid values: (manager | peer | searchhead | disabled) Defaults to disabled.Sets operational mode for this cluster node. Only one manager may exist per cluster. |
ping_flag | For internal use to facilitate communication between the manager and peers. | ||
quiet_period | The time, in seconds, that a manager waits for peers to add themselves to the cluster. | ||
rcv_timeout | Low-level timeout, in seconds, for receiving data between cluster nodes. Defaults to 60 seconds. | ||
register_forwarder_address | Not used.Reserved for future use. | ||
register_replication_address | Valid only for nodes configured as peers. The address on which a peer is available for accepting replication data. This is useful in the cases where a peer host machine has multiple interfaces and only one of them can be reached by another splunkd instance. | ||
register_search_address | IP address that advertises this indexer to search heads. | ||
rep_cxn_timeout | Low-level timeout, in seconds, for establishing a connection for replicating data. | ||
rep_max_rcv_timeout | Maximum cumulative time, in seconds, for receiving acknowledgement data from peers. Defaults to 600s. | ||
rep_max_send_timeout | Maximum time, in seconds, for sending replication slice data between cluster nodes. Defaults to 600s. | ||
rep_rcv_timeout | Low-level timeout, in seconds, for receiving data between cluster nodes. | ||
rep_send_timeout | Low-level timeout, in seconds, for sending replication data between cluster nodes. Defaults to 5 seconds. | ||
replication_factor | Only valid for nodes configured as a manager.Determines how many copies of raw data are created in the cluster. This could be less than the number of cluster peers.Must be greater than 0 and greater than or equal to the search factor. Defaults to 3. | ||
replication_port | TCP port to listen for replicated data from another cluster member. | ||
replication_use_ssl | Indicates whether to use SSL when sending replication data. | ||
restart_timeout | Only valid for nodes configured as a manager. The amount of time, in seconds, the manager waits for a peer to come back when the peer is restarted (to avoid the overhead of trying to fix the buckets that were on the peer). Defaults to 600 seconds.Note: This only works if the peer is restarted from Splunk Web. | ||
search_factor | Only valid for nodes configured as a manager. Determines how many searchable copies of each bucket to maintain. Must be less than or equal to replication_factor and greater than 0. Defaults to 2. | ||
secret | Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the manager, it is not able to communicate with the manager.Corresponds to pass4SymmKey setting in server.conf. | ||
send_timeout | Low-level timeout, in seconds, for sending data between cluster nodes. Defaults to 60 seconds. | ||
summary_replication | Boolean indicator of whether summary replication is on or off. A true value means that it is turned on. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/config
XML Response
clusterconfig https://localhost:8089/services/cluster/config 2012-09-05T10:19:49-07:00 Splunk ... opensearch nodes elided ... config https://localhost:8089/services/cluster/config/config 2012-09-05T10:19:49-07:00 system 60 0 ... eai:acl node elided ... 0 1 1 60 5 5 manager 1 60 60 5 600 600 10 5 2 0 600 2 ******** 60cluster/config/config
https://:/services/cluster/config/config
Manage cluster node configuration details.
GET
List cluster node configuration.
This operation works identically to the GET on cluster/config
.
POST
Manage configuration details.
See Indexer cluster configuration overview in Managing Indexers and Clusters of Indexers and the [clustering]
stanza options in the server.conf spec file for more details on indexer cluster configuration.
Request parameters
Name | Datatype | Description | ||
---|---|---|---|---|
available_sites | N/A | Sets the various sites that are recognized for this manager. Valid values include site1 to site64. | ||
cluster_label | String | Label for this cluster. | ||
cxn_timeout | Number | Low-level timeout, in seconds, for establishing connection between cluster nodes. Defaults to 60 seconds. | ||
heartbeat_period | Number | Only valid for peer nodes in a cluster. Time, in seconds, that a peer attempts to send a heartbeat to the manager | ||
heartbeat_timeout | Number | Only valid for the manager node in a cluster configuration. Time, in seconds, before a manager considers a peer down. Once a peer is down, the manager initiates steps to replicate buckets from the dead peer to its live peers. Defaults to 60 seconds. | ||
manager_uri | URI | Valid only for nodes configured as a peer or searchhead. URI of the cluster manager to which this node connects. | ||
max_peer_build_load | Number | The number of jobs that a peer can have in progress at any time that make the bucket searchable. | ||
max_peer_rep_load | Number | Maximum number of replications that can be ongoing as a target. | ||
mode | See description. | Required. Valid values: (manager | peer | searchhead | disabled) Defaults to disabled. Sets operational mode for this cluster node. Only one manager may exist per cluster. |
multisite | Boolean | Enable or disable the multisite feature for this cluster. | ||
notify_scan_period | Non-zero number | Controls the frequency that the indexer scans summary folders for summary updates. Only used when summary_replication is enabled on the manager. Defaults to 10 seconds. | ||
ping_flag | N/A | For internal use to facilitate communication between the manager and peers. | ||
quiet_period | Number | The time, in seconds, that a manager waits for peers to add themselves to the cluster. | ||
rcv_timeout | Number | Low-level timeout, in seconds, for receiving data between cluster nodes. Defaults to 60 seconds. | ||
register_forwarder_address | N/A | Reserved for future use. | ||
register_replication_address | See description. | Valid only for nodes configured as peers. The address on which a peer is available for accepting replication data. This is useful in the cases where a peer host machine has multiple interfaces and only one of them can be reached by another splunkd instance. | ||
register_search_address | N/A | IP address that advertises this indexer to search heads. | ||
rep_cxn_timeout | Number | Low-level timeout, in seconds, for establishing a connection for replicating data. | ||
rep_max_rcv_timeout | Number | Maximum cumulative time, in seconds, for receiving acknowledgement data from peers. Defaults to 600s. | ||
rep_max_send_timeout | Number | Maximum time, in seconds, for sending replication slice data between cluster nodes. Defaults to 600s. | ||
rep_rcv_timeout | Number | Low-level timeout, in seconds, for receiving data between cluster nodes. | ||
rep_send_timeout | Number | Low-level timeout, in seconds, for sending replication data between cluster nodes. Defaults to 5 seconds. | ||
replication_factor | Number | Only valid for nodes configured as a manager. Determines how many copies of raw data are created in the cluster. This could be less than the number of cluster peers. Must be greater than 0 and greater than or equal to the search factor. Defaults to 3. | ||
replication_port | Number | TCP port to listen for replicated data from another cluster member. | ||
replication_use_ssl | Number | Indicates whether to use SSL when sending replication data. | ||
restart_timeout | Number | Only valid for nodes configured as a manager. The amount of time, in seconds, the manager waits for a peer to come back when the peer is restarted (to avoid the overhead of trying to fix the buckets that were on the peer). Defaults to 600 seconds.Note: This only works if the peer is restarted from Splunk Web. | ||
search_factor | Number | Only valid for nodes configured as a manager. Determines how many searchable copies of each bucket to maintain. Must be less than or equal to replication_factor and greater than 0. Defaults to 2. | ||
secret | N/A | Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the manager, it is not able to communicate with the manager. Corresponds to pass4SymmKey setting in server.conf. | ||
send_timeout | Number | Low-level timeout, in seconds, for sending data between cluster nodes. Defaults to 60 seconds. | ||
site | N/A | Site ID for peer/searchhead indexer. Valid values include site1 to site64. | ||
site_replication_factor | Number | Replication factor for a multisite configuration. | ||
site_search_factor | Number | Search factor for a multisite configuration. | ||
summary_replication | Boolean | Enable or disable summary replication. | ||
use_batch_mask_changes | Boolean | Only valid for mode=manager .Specifies if the manager should process bucket mask changes inbatch or inidividually one by one. Defaults to true. Set to false when there are 6.1 peers in the cluster for backwards compatibility. |
Response data keys
None.
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/config/config -d cxn_timeout=59
XML Response
clusterconfig https://localhost:8089/services/cluster/config 2015-12-07T17:09:48-08:00 Splunk 0 30 0cluster/manager/buckets
https://:/services/cluster/manager/buckets
Provides bucket configuration information for a cluster manager node.
GET
List cluster manager node bucket configuration.
Request parameters
Use an &summaries=true
field in the query string to show summaries.
Use one or more filters in the query string to select buckets or bucket states. For example, use this URL to filter buckets returned for both the main
index and StreamingSource
status.
See the following table for available filters.
Filter name | Datatype | Description |
---|---|---|
index | String | Index name. |
status | String | Bucket state. Available options areStreamingSource StreamingTarget Complete StreamingError PendingTruncate Bucket is scheduled to truncate. PendingDiscard Bucket is scheduled to discard. NonStreamingTarget |
search_state | String | Bucket search state. Available options areSearchable Unsearchable PendingSearchable Bucket scheduled to become searchable by transferring or building tsidx files. PendingUnsearchable Bucket is scheduled to become unsearchable. SearchablePendingMask Primary change is scheduled or in progress. |
replication_count | Number | Use <, >, != or = with numbers to indicate filtering values. |
search_count | Number | Use <, >, != or = with numbers to indicate filtering values. |
bucket_size | Number | Use <, >, != or = with numbers to indicate filtering values. |
frozen | Boolean true | false | Return frozen buckets or non-frozen buckets. |
has_primary | Booleantrue | false | Return buckets with primaries or without primaries. |
meets_multisite_replication_count | Booleantrue | false | Return buckets that meet cluster replication policy or buckets that do not meet cluster replication policy. |
meets_multisite_search_count | Booleantrue | false | Return buckets that meet cluster search policy or buckets that do not meet cluster search policy. |
multisite_bucket | Booleantrue | false | Return buckets created in multisite mode or buckets not created in multisite mode. |
origin_site | String | Site of the indexer where buckets were created. |
standalone | Booleantrue | false | Use true or 1 to return standalone buckets. Use false or 0 to return clustered buckets. |
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
bucket_size | Indicates the size, in bytes, of the bucket. |
constrain_to_origin_site | Flag indicating this particular bucket is a clustered pre-multisite bucket. Such buckets are replicated only within their origin site. |
frozen | Indicates if the bucket is frozen. |
index | Name of the index to which the bucket belongs. |
origin_site | Where the bucket originated. |
peers | Lists information about buckets on peers to this manager. |
primaries_by_site | Primary peer (GIUD). |
rep_count_by_site | Number of buckets. |
search_count_by_site | Number of searchable buckets. |
service_after_time | Bucket service is deferred until after this time. |
standalone | Indicates if the bucket was created on the peer before the peer entered into a cluster configuration with this manager. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/buckets
XML Response
clustermanagerbuckets https://localhost:8089/services/cluster/manager/buckets 2014-04-17T19:13:57+00:00 Splunk 24 30 0 _audit~0~238C3311-F0A4-4A9B-97F0-53667CFFEEAB https://localhost:8089/services/cluster/manager/buckets/_audit~0~238C3311-F0A4-4A9B-97F0-53667CFFEEAB 2014-04-17T19:13:57+00:00 system 47187 1 ... elided ... 0 _audit site2 0x6 StableCksum Searchable Complete 0x0 StableCksum Searchable Complete 0x0 StableCksum Unsearchable Complete 238C3311-F0A4-4A9B-97F0-53667CFFEEAB 238C3311-F0A4-4A9B-97F0-53667CFFEEAB 3 2 0 0 . . . elided . . . _internal~1~E4B2C5E4-0961-4F3A-A5F7-C3A4BB6B518C https://localhost:8089/services/cluster/manager/buckets/_internal~1~E4B2C5E4-0961-4F3A-A5F7-C3A4BB6B518C 2014-04-17T19:13:57+00:00 system 0 ... elided ... 0 _internal site2 0x2 StableCksum Searchable StreamingTarget 0x0 StableCksum Unsearchable StreamingTarget 0x4 StableCksum Searchable StreamingSource 61666763-43E9-411B-9464-D80A5119EF0E E4B2C5E4-0961-4F3A-A5F7-C3A4BB6B518C 1 2 1 1 0 0cluster/manager/buckets/{name}
https://:/services/cluster/manager/buckets/{name}
Access bucket configuration information.
GET
List bucket configuration information.
Request parameters
The filter
parameter of the Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
bucket_size | Indicates the size, in bytes, of the bucket. |
constrain_to_origin_site | Flag indicating this particular bucket is a clustered pre-multisite bucket. Such buckets are replicated only within their origin site. |
frozen | Indicates if the bucket is frozen. |
index | Name of the index to which the bucket belongs. |
origin_site | Where the bucket originated. |
peers | Lists information about buckets on peers to this manager. |
primaries_by_site | Primary peer (GIUD). |
rep_count_by_site | Number of buckets. |
search_count_by_site | Number of searchable buckets. |
service_after_time | Bucket service is deferred until after this time. |
standalone | Indicates if the bucket was created on the peer before the peer entered into a cluster configuration with this manager. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/buckets/internal~1~238C3311-F0A4-4A9B-97F0-53667CFFEEAB
XML Response
clustermanagerbuckets https://localhost:8089/services/cluster/manager/buckets 2014-04-17T19:16:03+00:00 Splunk 1 30 0 _internal~1~238C3311-F0A4-4A9B-97F0-53667CFFEEAB https://localhost:8089/services/cluster/manager/buckets/_internal~1~238C3311-F0A4-4A9B-97F0-53667CFFEEAB 2014-04-17T19:16:03+00:00 system 0 ... elided ... ... elided ... 0 _internal site2 0x4 StableCksum Searchable StreamingSource 0x2 StableCksum Searchable StreamingTarget 0x0 StableCksum Unsearchable StreamingTarget 29F9560E-A44A-425C-8753-1C6158B46C84 238C3311-F0A4-4A9B-97F0-53667CFFEEAB 1 2 1 1 0 0cluster/manager/buckets/{bucket_id}/fix
https://:/services/cluster/manager/buckets/{bucket_id}/fix
Add the specified bucket to the fix list.
For more information, see Bucket-fixing scenarios in Managing Indexers and Clusters of Indexers.
Authentication and Authorization
Requires the admin
role or indexes_edit
capability.
POST
Add this bucket to the fix list.
Request parameters
None
Returned values
None
Example request and response
XML Request
curl -k -u admin:changeme https://localhost:8089/services/cluster/manager/buckets/_internal~0~111175BA-00DF-4CFE-9AEC-48A87B97EC71/fix -X POST
XML Response
clustermanagerbuckets https://localhost:8089/services/cluster/manager/buckets 2015-11-04T12:23:57-08:00 Splunk 0 30 0cluster/manager/buckets/{bucket_id}/fix_corrupt_bucket
https://:/services/cluster/manager/buckets/{bucket_id}/fix_corrupt_bucket
Trigger a corruption fixup of a clustered non-SmartStore-enabled bucket.
For more information, see Bucket-fixing scenarios in Managing Indexers and Clusters of Indexers.
Authentication and Authorization
Requires the admin
role or edit_indexer_cluster
capability.
POST
Trigger a corruption fixup for this bucket.
Request parameters
None
Returned values
None
Example request and response
XML Request
curl -k -u admin:changeme https://localhost:8089/services/cluster/manager/buckets/_internal~0~111175BA-00DF-4CFE-9AEC-48A87B97EC71/fix_corrupt_bucket -X POST
XML Response
"links":{ "create":"/services/cluster/manager/buckets/_new" }, "origin":"https://chieftain:15511/services/cluster/manager/buckets", "updated":"2023-09-06T22:30:08-07:00", "generator":{ "build":"479782058d4faa7ef3404e947f4117df3a59654c", "version":"20230905" }, "entry":[
], "paging":{ "total":0, "perPage":30, "offset":0 }, "messages":[
]
cluster/manager/buckets/{bucket_id}/freeze
https://:/services/cluster/manager/buckets/{bucket_id}/freeze
Set the bucket's state to frozen. The frozen state may not persist after a cluster manager restart unless one of the peers has set the frozen state. A POST to this endpoint does not set the bucket's state to frozen on peers.
Note: Use this endpoint with caution. It is recommended to test the endpoint in a test cluster prior to use on an actual bucket.
For more information, see How the cluster handles frozen buckets in Managing Indexers and Clusters of Indexers.
Authentication and Authorization
Requires the admin
role or indexes_edit
capability.
POST
Set this bucket's state to frozen.
Request parameters
None
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/buckets/_internal~0~111175BA-00DF-4CFE-9AEC-48A87B97EC71/freeze -X POST
XML Response
clustermanagerbuckets https://locahost:8089/services/cluster/manager/buckets 2015-11-04T12:21:27-08:00 Splunk 0 30 0cluster/manager/buckets/{bucket_id}/remove_all
https://:/services/cluster/manager/buckets/{bucket_id}/remove_all
Delete all copies of the specified bucket.
Caution: Using this endpoint will cause irreversible data loss. It is recommended to test the endpoint on a test cluster prior to use on an actual bucket.
Authentication and Authorization
Requires the admin
role or indexes_edit
capability.
POST
Delete all copies of the specified bucket.
Request parameters
None
Returned values
None. If an invalid bucket id is used, an error message is returned.
In handler 'clustermanagerbuckets': bucket not foundIf the request is made on a hot bucket, an error message is returned.
In handler 'clustermanagerbuckets': cannot remove hot bucket from clusterExample request and response
XML Request
curl -k -u admin:password https://localhost:8089/services/cluster/manager/buckets/_internal~0~111175BA-00DF-4CFE-9AEC-48A87B97EC71/remove_all -X POST
XML Response
clustermanagerbuckets https://localhost:8089/services/cluster/manager/buckets 2015-11-04T12:24:12-08:00 Splunk 0 30 0cluster/manager/buckets/{bucket_id}/remove_from_peer
https://:/services/cluster/manager/buckets/{bucket_id}/remove_from_peer
Deletes the copy of this bucket from specified peer.
If the request causes the cluster to lose its complete state, the cluster will engage in fixup activities. This may result in another copy of the same bucket appearing on this peer. If, however, the specified bucket is frozen, the cluster does not attempt any fixup activities.
Caution: Using this endpoint will cause irreversible data loss. It is recommended to test the endpoint on a test-cluster prior to use on an actual bucket.
Authentication and Authorization
Requires the admin
role or indexes_edit
capability.
POST
Delete this bucket from specified peer. Set bucket state to frozen
Request parameters
Name | Type | Description |
---|---|---|
peer (required) | GUID | Peer GUID |
Returned values
None. If the peer
parameter is missing from the request, an error message is returned.
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/buckets/_internal~0~111175BA-00DF-4CFE-9AEC-48A87B97EC71/remove_from_peer -X POST -d peer=222275BA-00DF-4CFE-9AEC-48A87B97EC71
XML Response
clustermanagerbuckets https://localhost:8089/services/cluster/manager/buckets 2015-11-04T12:23:18-08:00 Splunk 0 30 0cluster/manager/control/control/prune_index
https://:/services/cluster/manager/control/control/prune_index
Clean up excess bucket copies across an index.
For more information, see Remove extra bucket copies in Managing Indexers and Clusters of Indexers.
POST
Clean up excess bucket copies across an index.
Request parameters
Name | Description |
---|---|
index | Optional. The index from which to remove excess bucket copies. If not specified, the POST operation clears excess bucket copies across all indexes. |
Returned values
None
Example request
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/control/control/prune_index -d index="my_index"
cluster/manager/control/control/rebalance_primaries
https://:/services/cluster/manager/control/control/rebalance_primaries
Rebalance primary bucket copies across peers. For more information, see Rebalance the indexer cluster primary buckets in Managing Indexers and Clusters of Indexers.
POST
Rebalance primary buckets across all peers of this manager.
Request parameters
None
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/control/control/rebalance_primaries --request POST
XML Response
clustermanagercontrol https://localhost:8089/services/cluster/manager/control 2013-08-21T13:08:52-07:00 Splunk 0 30 0cluster/manager/control/control/remove_peers
https://:/services/cluster/manager/control/control/remove_peers
Remove one or more peers.
See also
cluster/manager/peers
POST
Remove one or more peers.
Request parameters
Name | Type | Description |
---|---|---|
peers Required | String | One or more comma-separated peer GUIDs. |
Returned values
None
Application usage
If peer status is not Down or GracefulShutdown, the interface returns the following error message:
In handler 'clustermanagercontrol': Remove aborted, Reason: Peer= with guid= cannot be removed. Peer has status=Up. Only peers with status=Down (or) GracefulShutdown can be removed.Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/control/control/remove_peers --request POST -d "peers=F2AA19BD-622F-4F8C-A8E0-1233"
XML Response
clustermanagercontrol https://localhost:8089/services/cluster/manager/control 2014-09-10T13:12:54-07:00 Splunk ... opensearch nodes elided ...cluster/manager/control/control/resync_bucket_from_peer
https://:/services/cluster/manager/control/control/resync_bucket_from_peer
This endpoint resets the state of a specified bucket based on the current state of the bucket at a peer.
POST
Reset bucket state based on the current state of the bucket at a peer.
Request Parameters
| Name | Type | Default | Description | | | ------------ | ------ | ------- | ----------------------------------------------------------- | | | bucket_id | String | N/A | Required. ID of bucket to update. | | | peer | GUID | N/A | Required. GUID of peer from which to update the bucket. | |
Returned Values
None.
Example request and response
XML Request
curl -k -u admin:pass https://hostname:mPort:/services/cluster/manager/control/control/resync_bucket_from_peer -X POST -d bucket_id=_audit28F6747E9-88C9-4488-8806-4EA3CA433CF5 -d peer=8F6747E9-88C9-4488-8806-4EA3CA433CF5
XML Response
clustermanagercontrol https://10.66.129.225:8089/services/cluster/manager/control 2016-06-30T14:32:06+08:00 Splunk 0 30 0cluster/manager/control/control/roll-hot-buckets
https://:/services/cluster/manager/control/control/roll-hot-buckets
This endpoint forces a specified bucket in an indexer cluster to roll from hot to warm. Pass the bucket id (bid) to the manager node. The manager instructs the origin peer for that bucket to roll its copy. In turn, the origin peer tells all the replicating peers to roll their copies
You might discover a bucket that is stuck in fixup and needs to be rolled using logs, Splunk Web, or either of the following two endpoints.
Authorization and authentication
This endpoint requires the admin role for use.
POST
Force a bucket to roll from hot to warm.
Request parameters
Name | Type | Default | Description |
---|---|---|---|
bucket_id | String | N/A | Required. ID for bucket to roll. |
Returned values
None.
Example request and response
XML Request
curl -k -u username:password https://localhost:8089/services/cluster/manager/control/control/roll-hot-buckets -X POST -d "bucket_id=_audit21A3889D7-954B-4CE6-B071-01B438DE9865"
XML Response
clustermanagercontrol https://localhost:8089/services/cluster/manager/control 2015-10-30T07:34:56+08:00 Splunk 0 30 0cluster/manager/control/control/rolling_upgrade_finalize
https://:/services/cluster/manager/control/control/rolling_upgrade_finalize
Finalizes an indexer cluster rolling upgrade.
POST
Finalizes an indexer cluster rolling upgrade.
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/control/control/rolling_upgrade_finalize -X POST
XML Response
clustermanagercontrol https://10.141.65.179:52000/services/cluster/manager/control 2018-04-01T22:04:46+00:00 Splunk 0 30 0 Cluster is no longer in searchable rolling upgrade mode.cluster/manager/control/control/rolling_upgrade_init
https://:/services/cluster/manager/control/control/rolling_upgrade_init
Initializes an indexer cluster rolling upgrade.
POST
Initializes an indexer cluster rolling upgrade.
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/control/control/rolling_upgrade_init -X POST
XML Response
clustermanagercontrol https://10.141.65.179:52000/services/cluster/manager/control 2018-04-01T21:06:21+00:00 Splunk 0 30 0 Cluster is now in searchable rolling upgrade mode.cluster/manager/control/default/abort_restart
https://:/services/cluster/manager/control/default/abort_restart
Aborts an ongoing restart of an indexer cluster.
Authentication and Authorization
Requires the admin
role or edit_indexer_cluster
capability.
POST
Abort an ongoing restart of an indexer cluster.
Request parameters
None
Returned values
None
Example request and response
JSON Request
curl -k -u admin:password -X POST "https://chieftain:15511/services/cluster/manager/control/default/abort_restart?output_mode=json"
JSON Response
{ "links":{
}, "origin":"https://chieftain:15511/services/cluster/manager/control", "updated":"2023-09-06T23:45:53-07:00", "generator":{ "build":"479782058d4faa7ef3404e947f4117df3a59654c", "version":"20230905" }, "entry":[
], "paging":{ "total":0, "perPage":30, "offset":0 }, "messages":[ { "type":"INFO", "text":"Aborting the rolling restart initiated successfully. List of peers skipped restarting: E30CA8C0-23E5-4A6B-9F28-D2EC991CCD75,9E3FED8B-59A0-4B95-8116-F8F8A67A7686,32790C7F-82CB-4E39-8689-3600F72D4D01,2B6C57ED-9FFC-44F0-9E58-CD8BE3519F3F,5A65CEB6-79A6-40D7-914C-4859DEACF79B,8C2DC775-EB8E-44D7-AFF8-38482B3A9990,033085C7-F31B-467D-9577-B8A5E5131810" } ] }
cluster/manager/control/default/apply
https://:/services/cluster/manager/control/default/apply
Pushes a bundle.
POST
Push a bundle.
Request Parameters
Name | Type | Default | Description |
---|---|---|---|
skip-validation | Boolean | False | Set as true to skip the validation step for this bundle. |
ignore_identical_bundle | Boolean | True | Set as false to push this bundle even if current active bundle is identical to this bundle. |
Returned Values
None.
Example request and response
XML Request
curl -k -u admin:pass https://host:mPort/services/cluster/manager/control/default/apply -X POST
XML Response
clustermanagercontrol https://wimpy:7420/services/cluster/manager/control 2019-01-02T13:46:04-08:00 Splunk 1 30 0 clusterbundles https://wimpy:7420/services/cluster/manager/control/clusterbundles 1969-12-31T16:00:00-08:00 system 288845778D5B1952F534AB16DD82881E 1 1 0 system admin splunk-system-role admin splunk-system-role 0 systemcluster/manager/control/default/cancel_bundle_push
https://:/services/cluster/manager/control/default/cancel_bundle_push
Cancels and resets the bundle push operation. Use this endpoint when the cluster manager does not receive a validation response from the cluster peer due to an error. For more information, see Configuration bundle issues.
POST
Cancel and reset the bundle push operation.
Request Parameters
None.
Returned Values
None.
Example request and response
XML Request
curl -k -u admin:pass https://host:mPort/services/cluster/manager/control/default/cancel_bundle_push -X POST
XML Response
clustermanagercontrol https:/:/services/cluster/manager/control 2017-08-21T15:13:13-07:00 Splunk 0 30 0cluster/manager/control/default/maintenance
https://:/services/cluster/manager/control/default/maintenance
Put the cluster manager into maintenance mode.
POST
Toggle maintenance mode.
Request Parameters
Name | Datatype | Description |
---|---|---|
mode | Boolean | Enable or disable maintenance mode on the cluster manager. |
Returned Values
None.
Example request and response
XML Request
curl -k -u username:pass https://:/services/cluster/manager/control/default/maintenance -d mode=true
XML Response
clustermanagercontrol https://myserver:8089/services/cluster/manager/control 2020-05-15T05:45:49+00:00 Splunk 0 30 0cluster/manager/control/default/rollback
https://:/services/cluster/manager/control/default/rollback
Roll a bundle back to the previously active bundle.
POST
Roll back a bundle.
Request Parameters
None.
Returned Values
None.
Example request and response
XML Request
curl -k -u admin:pass https://host:mPort/services/cluster/manager/control/default/rollback -X POST
XML Response
clustermanagercontrol https://wimpy:7420/services/cluster/manager/control 2019-01-02T13:46:26-08:00 Splunk 1 30 0 clusterbundles https://wimpy:7420/services/cluster/manager/control/clusterbundles 1969-12-31T16:00:00-08:00 system 447F196DB0CF55389029A950E3C2D3E3 1 1 0 system admin splunk-system-role admin splunk-system-role 0 systemcluster/manager/control/default/validate_bundle
https://:/services/cluster/manager/control/default/validate_bundle
Tests if the bundle in etc/manager-apps
passes validation. Optionally, tests if the bundle will trigger an indexer restart.
POST
Validate a bundle.
Request Parameters
Name | Type | Default | Description |
---|---|---|---|
check-restart | Boolean | False | By default, checks if the bundle passes validation on the cluster manager and indexers. Set to true to check if the bundle will trigger a restart on the indexers. |
Returned Values
None.
Example request and response
XML Request
curl -k -u admin:pass https://host:mPort/services/cluster/manager/control/default/validate_bundle -d check-restart=true -X POST
XML Response
clustermanagercontrol https://wimpy:7420/services/cluster/manager/control 2019-01-02T13:56:48-08:00 Splunk 1 30 0 clusterbundles https://wimpy:7420/services/cluster/manager/control/clusterbundles 1969-12-31T16:00:00-08:00 system 288845778D5B1952F534AB16DD82881E 1 1 0 system admin splunk-system-role admin splunk-system-role 0 systemcluster/manager/fixup
https://:/services/cluster/manager/fixup
Access a list of buckets on a specific fixup priority level. Bucket fixups are processed in order of priority level. See Request parameters below for priority level details.
When you access a particular fixup level, buckets may appear in it even though they do not need fixup at this level. Initially, each bucket requiring fixup is added to all levels, even though it might only require processing in a subset of all levels. As the bucket is processed through a level, it is deleted from that level.
GET
List buckets on the specified fixup level.
Request parameters
Pagination and filtering parameters can be used with this method.
Name | Datatype | Description |
---|---|---|
level | String | Required. Fixup priority level. Use one of the following level values, listed in order of priority.corruption : Corrupted buckets. streaming : Hot buckets that need to be rolled or have their size committed. data_safety : Buckets without at least two rawdata copies. generation : Buckets without a primary copy. replication_factor : Buckets without replication factor number of copies. search_factor : Buckets without search factor number of copies. checksum_sync : Level for syncing a bucket's delete files across all peers that have this bucket. Syncing is determined based on the checksum of all of the delete files. |
index | String | Optional. Index name. |
Returned values
For each bucket in the specified fixup level, the response includes the following details for the initial
time when the bucket went into the fixup level and the latest
time that the bucket was checked.
Name | Description |
---|---|
id | Bucket id. |
reason | Initial or latest reason for the bucket being on this fixup level. |
timestamp | Timestamp for initial bucket addition to fixup list or latest bucket check. |
Example request and response
XML Request
curl -k -u admin:password https://localhost:8089/services/cluster/manager/fixup?level=replication_factor
XML Response
clustermanagerfixup https://localhost:8089/services/cluster/manager/fixup 2015-11-09T17:05:48-08:00 Splunk 2 30 0 _audit~212~22220097-5E3F-4D26-B301-ECE3C4CD2222 https://localhost:8089/services/cluster/manager/fixup/_audit~212~22220097-5E3F-4D26-B301-ECE3C4CD2222 2015-11-09T17:05:48-08:00 system 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system _audit add peer=22220097-5E3F-4D26-B301-ECE3C4CD2222 new bucket 1447099323 Missing enough suitable candidates to create replicated copy in order to meet replication policy. Missing={ site2:1 } 1447117547 replication_factor _internal~12628~111163F8-61F4-4AB3-A1A7-2EDCB10C1111 https://localhost:8089/services/cluster/manager/fixup/_internal~12628~111163F8-61F4-4AB3-A1A7-2EDCB10C1111 2015-11-09T17:05:48-08:00 system 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system _internal add peer=111163F8-61F4-4AB3-A1A7-2EDCB10C1111 new bucket 1447099323 Missing enough suitable candidates to create replicated copy in order to meet replication policy. Missing={ site1:1 } 1447117547 replication_factorcluster/manager/generation
https://:/services/cluster/manager/generation
Access current generation cluster manager information and create a cluster generation.
GET
List peer nodes participating in the current generation for this manager.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
generation_id | The ID for the current generation for this manager. |
generation_peers | Lists the peers for this generation of the cluster. |
pending_generation_id | The next generation ID used by the manager when committing a new generation.This value is useful for debugging. |
pending_last_attempt | The timestamp of the last attempt to commit to the pending generation ID (if ever). |
pending_last_reason | The reason why this peer failed to commit to the pending generation.This parameter is EMPTY if no such attempt was made. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/generation
XML Response
clustermanagergeneration https://localhost:8089/services/cluster/manager/generation 2012-09-05T10:39:54-07:00 Splunk ... opensearch nodes elided ... manager https://localhost:8089/services/cluster/manager/generation/manager 2012-09-05T10:39:54-07:00 system ... eai:acl node elided ... 2 splunks-ombra.sv.splunk.com:8389 splunks-ombra.sv.splunk.com splunks-ombra.sv.splunk.com:8189 splunks-ombra.sv.splunk.com 3 0POST
Create a cluster generation.
Request parameters
Name | Type | Default | Description |
---|---|---|---|
_name_required | String | The URI of the searchhead node of a cluster upon which to create a new generation. | |
generation_poll_interval | Number | How often, in seconds, the searchhead polls the manager for generation information.Defaults to 60 seconds. | |
label | String | Server name for the Splunk platform instance specified by the name attribute. | |
mgmt_port | String | The managment port of searchhead node in a cluster upon which you are creating a new generation. | |
register_search_address | String | The address on which a peer node is available as search head.This is useful in the cases where a host machine has multiple interfaces and only one of them can be reached by another splunkd instance. |
Returned values
Name | Description |
---|---|
generation_id | The ID for the current generation for this manager. |
generation_peers | Lists the peers for this generation of the cluster. |
pending_generation_id | The next generation ID used by the manager when committing a new generation.This value is useful for debugging. |
pending_last_attempt | The timestamp of the last attempt to commit to the pending generation ID (if ever). |
pending_last_reason | The reason why this peer failed to commit to the pending generation.This parameter is EMPTY if no such attempt was made. |
replication_factor_met | Indicates if the replication factor was met for the cluster. |
search_factor_met | Indicates if the search factor was met for the cluster. |
was_forced | Indicates next generation was forcibly committed. |
Example request and response
XML Request
curl -k -u admin:pass https://myserver:8089/services/cluster/manager/generation -d name=foo
XML Response
clustermanagergeneration https://myserver:8089/services/cluster/manager/generation 2013-10-31T13:58:51-07:00 Splunk ... opensearch nodes elided ... manager https://myserver:8089/services/cluster/manager/generation/manager 2013-10-31T13:58:51-07:00 system ... eai:acl node elided ... 5 myserver.splunk.com:6431 PEER1 myserver.splunk.com:6432 PEER2 myserver.splunk.com:6433 PEER3 myserver.splunk.com:6434 PEER4 6 0 1 1 0cluster/manager/generation/{name}
https://:/services/cluster/manager/generation/{name}
Access information about a peer node participating in the current generation for the specified search head GUID.
GET
List peer node information of the specified search head GUID.
Request parameters
None
Returned values
Name | Description |
---|---|
generation_id | The ID of the current generation for this manager. |
generation_peers | Lists the peers for this generation of the cluster. |
pending_generation_id | The next generation ID used by the manager when committing a new generation.This value is useful for debugging. |
pending_last_attempt | The timestamp of the last attempt to commit to the pending generation ID (if ever). |
pending_last_reason | The reason why this peer failed to commit to the pending generation.This parameter is EMPTY if no such attempt was made. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/generation/manager
XML Response
clustermanagergeneration https://localhost:8089/services/cluster/manager/generation 2012-09-05T10:45:27-07:00 Splunk ... opensearch nodes elided ... manager https://localhost:8089/services/cluster/manager/generation/manager 2012-09-05T10:45:27-07:00 system ... eai:acl node elided ... ... eai:attributes node elided ... 2 splunks-ombra.sv.splunk.com:8389 splunks-ombra.sv.splunk.com splunks-ombra.sv.splunk.com:8189 splunks-ombra.sv.splunk.com 3 0POST
Create a new generation for the specified search head GUID.
Request parameters
Name | Type | Description |
---|---|---|
generation_poll_interval | Number | How often, in seconds, the searchhead polls the manager for generation information.Defaults to 60 seconds. |
label | String | Server name for the search head specified by {name}. |
mgmt_port | String | The managment port of searchhead node in a cluster upon which you are creating a new generation. |
register_search_address | String | The address on which a peer node is available as search head.This is useful when a host machine has multiple interfaces and only one of them can be reached by another splunkd instance. |
Returned values
Name | Description |
---|---|
generation_id | The ID for the current generation for this manager. |
generation_peers | Lists the peers for this generation of the cluster. |
pending_generation_id | The next generation ID used by the manager when committing a new generation.This value is useful for debugging. |
pending_last_attempt | The timestamp of the last attempt to commit to the pending generation ID (if ever). |
pending_last_reason | The reason why this peer failed to commit to the pending generation.This parameter is EMPTY if no such attempt was made. |
replication_factor_met | Indicates if the replication factor was met for the cluster. |
search_factor_met | Indicates if the search factor was met for the cluster. |
was_forced | Indicates next generation was forcibly committed. |
Example request and response
XML Request
curl -k -u admin:pass https://myserver:8089/services/cluster/manager/generation/foo -X POST -d generation_poll_interval=62 -d label=PEER2
XML Response
clustermanagergeneration https://myserver:8089/services/cluster/manager/generation 2013-10-31T14:37:20-07:00 Splunk ... opensearch nodes elided ... manager https://myserver:8089/services/cluster/manager/generation/manager 2013-10-31T14:37:20-07:00 system ... eai:acl node elided ... 5 myserver.splunk.com:6431 PEER1 myserver.splunk.com:6432 PEER2 myserver.splunk.com:6433 PEER3 myserver.splunk.com:6434 PEER4 6 0 1 1 0cluster/manager/ha_active_status
https://:/services/cluster/manager/ha_active_status
Used by the load balancers to check the high availability mode of a given cluster manager.
The active cluster manager will return "HTTP 200", denoting "healthy", and a startup or standby cluster manager will return "HTTP 503".
Authentication and authorization
This endpoint is unauthenticated because some load balancers don't support authentication on a health check endpoint.
GET
Checks the high availability mode of a given cluster manager.
Request parameters
None
Returned values
None
Example request and response
Request
curl -k -v -u admin:changeme https://mrt:15511/services/cluster/manager/ha_active_status
Response
From active cluster manager:
< HTTP/1.1 200 OK < Date: Tue, 10 May 2022 10:45:57 GMT < Expires: Thu, 26 Oct 1978 00:00:00 GMT < Cache-Control: no-store, no-cache, must-revalidate, max-age=0 < Content-Type: text/xml; charset=UTF-8 < X-Content-Type-Options: nosniff < Content-Length: 1740 < Connection: Keep-Alive < X-Frame-Options: SAMEORIGIN < Server: Splunkd <
clusteractivemanager https://mrt:15511/services/cluster/manager/ha_active_status 2022-05-10T10:45:57+00:00 Splunk 0 30 0From standby cluster manager:
< HTTP/1.1 503 Service Unavailable < Date: Tue, 10 May 2022 10:47:00 GMT < Expires: Thu, 26 Oct 1978 00:00:00 GMT < Cache-Control: no-store, no-cache, must-revalidate, max-age=0 < Content-Type: text/xml; charset=UTF-8 < X-Content-Type-Options: nosniff < Content-Length: 154 < Connection: Keep-Alive < X-Frame-Options: SAMEORIGIN < Server: Splunkd < Cluster manager is in inactive mode.
cluster/manager/health
https://:/services/cluster/manager/health
Performs health checks to determine the cluster health and search impact, prior to a rolling upgrade of the indexer cluster.
Authentication and Authorization
Requires the admin
role or list_indexer_cluster
capability.
GET
Get indexer cluster health check results.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Datatype | Description |
---|---|---|
all_data_is_searchable | Boolean | Indicates if all data in the cluster is searchable. |
all_peers_are_up | Boolean | Indicate if all peers are strictly in the Up status. |
cm_version_is_compatible | Boolean | Indicates if any cluster peers are running a Splunk Enterprise version greater than or equal to the cluster manager's version. |
multisite | Boolean | Indicates if multisite is enabled. |
no_fixups_in_progress | Boolean | Indicates if there does not exist buckets with bucket state NonStreamingTarget, or bucket search states PendingSearchable or SearchablePendingMask. |
pre_flight_check | Boolean | Indicates if the health check prior to a rolling upgrade was successful. This value is true only if the cluster passed all health checks. |
replication_factor_met | Boolean | Only valid for mode=manager and multisite=false. Indicates whether the replication factor is met. If true, the cluster has at least replication_factor number of raw data copies in the cluster. |
search_factor_met | Boolean | Only valid for mode=manager and multisite=false. Indicates whether the search factor is met. If true, the cluster has at least search_factor number of raw data copies in the cluster. |
site_replication_factor_met | Boolean | Only valid for mode=manager and multisite=true. Indicates whether the site replication factor is met. If true, the cluster has at least replication_factor number of raw data copies in the cluster. |
site_search_factor_met | Boolean | Only valid for mode=manager and multisite=true. Indicates whether the site search factor is met. If true, the cluster has at least site_search_factor number of raw data copies in the cluster. |
splunk_version_peer_count | String | Lists the number of cluster peers running each Splunk Enterprise version. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/health
XML Response
clustermanagerhealth https://10.141.65.179:52000/services/cluster/manager/health 2018-04-01T19:53:47+00:00 Splunk 1 30 0 manager https://10.141.65.179:52000/services/cluster/manager/health/manager 1970-01-01T00:00:00+00:00 system 1 1 1 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system 0 1 1 1 1 1 1 { 7.1.0: 3 }cluster/manager/indexes
https://:/services/cluster/manager/indexes
Access cluster index information.
GET
List cluster indices.
Request parameters
None
Returned values
Attribute | Description |
---|---|
buckets_with_excess_copies | Number of distinct buckets that have one or more excess replication copies. |
buckets_with_excess_searchable_copies | Number of distinct buckets that have one or more excess searchable copies. |
index_size | Size of the index |
is_searchable | When every bucket in the index has a primary, the index is considered "searchable". |
non_site_aware_buckets_in_site_aware_cluster | Number of buckets created when the cluster was not in a multisite config. (Included only when the cluster is in multisite config.) |
num_buckets | Total number of distinct buckets. |
replicated_copies_tracker | Displays how many distinct buckets have X number of copies. One of the following options.actual_copies_per_slot Number of buckets with X copies. expected_total_per_slot Expected number of buckets with X copies. |
searchable_copies_tracker | Displays how many distinct buckets have X number of searchable copies. One of the following options.actual_copies_per_slot Number of buckets with X searchable copies. expected_total_per_slot Expected number of buckets with X searchable copies. |
sort_order | Used by UI. |
total_excess_bucket_copies | Total number of excess copies for all buckets. |
total_excess_searchable_copies | Total number of excess searchable copies for all buckets. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/indexes
XML Response
clustermanagerpeerindexes https://localhost:8089/services/cluster/manager/indexes 2014-04-17T19:11:14+00:00 Splunk 2 30 0 _audit https://localhost:8089/services/cluster/manager/indexes/_audit 2014-04-17T19:11:14+00:00 system 0 0 ... elided ... 284975 1 6 12 12 12 12 12 12 12 12 12 12 12 4294967295 0 0 _internal https://localhost:8089/services/cluster/manager/indexes/_internal 2014-04-17T19:11:14+00:00 system 0 0 ... elided ... 1190869 1 6 12 12 12 12 12 12 12 12 12 12 12 4294967295 0 0cluster/manager/indexes/{name}
https://:/services/cluster/manager/indexes/{name}
Access specific cluster index information.
GET
List {name} index information.
Request parameters
None
Returned values
Attribute | Description |
---|---|
buckets_with_excess_copies | Number of distinct buckets that have one or more excess replication copies. |
buckets_with_excess_searchable_copies | Number of distinct buckets that have one or more excess searchable copies. |
index_size | Size of the index |
is_searchable | When every bucket in the index has a primary, the index is considered "searchable". |
non_site_aware_buckets_in_site_aware_cluster | Number of buckets created when the cluster was not in a multisite config. (Included only when the cluster is in multisite config.) |
num_buckets | Total number of distinct buckets. Displays how many distinct buckets have X number of copies. One of the following options.actual_copies_per_slot Number of buckets with X copies. expected_total_per_slot Expected number of buckets with X copies. |
searchable_copies_tracker | Displays how many distinct buckets have X number of searchable copies. One of the following options.actual_copies_per_slot Number of buckets with X searchable copies. expected_total_per_slot Expected number of buckets with X searchable copies. |
sort_order | Used by UI. |
total_excess_bucket_copies | Total number of excess copies for all buckets. |
total_excess_searchable_copies | Total number of excess searchable copies for all buckets. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/indexes/_audit
XML Response
clustermanagerpeerindexes https://localhost:8089/services/cluster/manager/indexes 2014-04-17T19:11:14+00:00 Splunk 2 30 0 _audit https://localhost:8089/services/cluster/manager/indexes/_audit 2014-04-17T19:11:14+00:00 system 0 0 ... elided ... 284975 1 6 12 12 12 12 12 12 12 12 12 12 12 4294967295 0 0cluster/manager/info
https://:/services/cluster/manager/info
Access information about cluster manager node.
GET
List cluster manager node details.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
active_bundle | Provides information about the active bundle for this manager. |
bundle_creation_time_on_manager | The time, in epoch seconds, when the bundle was created on the manager. |
bundle_validation_errors_on_manager | A list of bundle validation errors. |
bundle_validation_in_progress | Indicates if bundle validation is in progress. |
bundle_validation_on_manager_succeeded | Indicates whether the manager succeeded validating bundles. |
data_safety_buckets_to_fix | Lists the buckets to fix for the completion of data safety. |
gen_commit_buckets_to_fix | The buckets to be fixed before the next generation can be committed. |
indexing_ready_flag | Indicates if the cluster is ready for indexing. |
initialized_flag | Indicates if the cluster is initialized. |
label | The name for the manager. Displayed in the Splunk Web manager page. |
latest_bundle | The most recent information reflecting any changes made to the manager-apps configuration bundle.In steady state, this is equal to active_bundle. If it is not equal, then pushing the latest bundle to all peers is in process (or needs to be started). |
maintenance_mode | Indicates if the cluster is in maintenance mode. |
reload_bundle_issued | Indicates if the bundle issued is being reloaded. |
rep_count_buckets_to_fix | Number of buckets to fix on peers. |
rolling_restart_flag | Indicates whether the manager is restarting the peers in a cluster. |
search_count_buckets_to_fix | Number of buckets to fix to satisfy the search count. |
service_ready_flag | Indicates whether the manager is ready to begin servicing, based on whether it is initialized. |
start_time | Timestamp corresponding to the creation of the manager. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/info
XML Response
clustermanagerinfo http://greentea.sv.splunk.com:8089/services/cluster/manager/info 2013-07-23T10:36:35-07:00 Splunk ... opensearch nodes elided ... manager http://greentea.sv.splunk.com:8089/services/cluster/manager/info/manager 2013-07-23T10:36:35-07:00 system /home/eserv/schoi/apple_manager/splunk/var/run/splunk/cluster/remote-bundle/66e383cafa8ff1f033e2341e35fc2e09-1374594357.bundle a98f211c7bc6b141bd4fe5775c7cd193 1374594357 1374594357 0 1 resolved initial state 1374594631 1374600995 resolved initial state 1374594631 1374600995 . . . elided . . . streaming success 1374600995 streaming success 1374600995 ... eai:acl node elided ... resolved initial state 1374594631 1374600995 resolved initial state 1374594631 1374600995 . . . elided . . . streaming success 1374600995 streaming success 1374600995 1 1 manager_nc /home/eserv/schoi/apple_manager/splunk/var/run/splunk/cluster/remote-bundle/66e383cafa8ff1f033e2341e35fc2e09-1374594357.bundle a98f211c7bc6b141bd4fe5775c7cd193 1374594357 0 0 resolved initial state 1374594631 1374600995 resolved initial state 1374594631 1374600995 . . . elided . . . streaming success 1374600995 streaming success 1374600995 0 resolved initial state 1374594631 1374600995 resolved initial state 1374594631 1374600995 . . . elided . . . streaming success 1374600995 streaming success 1374600995 1 1374594571cluster/manager/peers
https://:/services/cluster/manager/peers
Access cluster manager peers.
See also
cluster/manager/control/control/remove_peers
GET
List cluster manager peers.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
active_bundle_id | The ID of the configuration bundle currently being used by the manager. |
apply_bundle_status | Bundle status enumeration. |
base_generation_id | The initial bundle generation ID recognized by this peer. Any searches from previous generations fail.The initial bundle generation ID is created when a peer first comes online, restarts, or recontacts the manager. |
bucket_count | Count of the number of buckets on this peer, across all indexes. |
bucket_count_by_index | Count of the number of buckets by index on this peer. |
delayed_buckets_to_discard | List of bucket IDs waiting to be discarded on this peer. |
fixup_set | The set of buckets that need repair once you take the peer offline. |
heartbeat_started | Flag indicating if this peer has started heartbeating. |
host_port_pair | The host and port advertised to peers for the data replication channel.Can be either of the form IP:port or hostname:port. |
is_searchable | Flag indicating if this peer belongs to the current committed generation and is searchable. |
label | The name for the peer. Displayed on the manager page. |
last_heartbeat | Timestamp for last heartbeat recieved from the peer. |
latest_bundle_id | The ID of the configuration bundle this peer is using. |
pending_job_count | Used by the manager to keep track of pending jobs requested by the manager to this peer. |
primary_count | Number of buckets for which the peer is primary in its local site, or the number of buckets that return search results from same site as the peer. |
primary_count_remote | Number of buckets for which the peer is primary that are not in its local site. |
replication_count | Number of replications this peer is part of, as either source or target. |
replication_port | TCP port to listen for replicated data from another cluster member. |
replication_use_ssl | Indicates whether to use SSL when sending replication data. |
search_state_counter | Lists the number of buckets on the peer for each search state for the bucket.Possible values for search state include: Searchable Unsearchable |
site | To which site the peer belongs. |
status | Indicates the status of the peer.Valid values are: Up Pending AutomaticDetention ManualDetention-PortsEnabled ManualDetention Restarting ShuttingDown ReassigningPrimaries Decommissioning GracefulShutdown Stopped Down BatchAdding |
status_counter | Lists the number of buckets on the peer for each bucket status.Possible values for bucket status: Complete: complete (warm/cold) bucket NonStreamingTarget: target of replication for already completed (warm/cold) bucket PendingTruncate: bucket pending truncation PendingDiscard: bucket pending discard Standalone: bucket that is not replicated StreamingError: copy of streaming bucket where some error was encountered StreamingSource: streaming hot bucket on source side StreamingTarget: streaming hot bucket copy on target side Unset: uninitialized |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/peers
XML Response
clustermanagerpeers https://localhost:8089/services/cluster/manager/peers 2014-04-17T19:17:08+00:00 Splunk 6 30 0 238C3311-F0A4-4A9B-97F0-53667CFFEEAB https://localhost:8089/services/cluster/manager/peers/238C3311-F0A4-4A9B-97F0-53667CFFEEAB 2014-04-17T19:17:08+00:00 system 4708B74780A1E5101449548B1E103616 0 6 10 5 5 ... elided ... 1 127.0.1.1:8096 1 s2p3 1397762228 4708B74780A1E5101449548B1E103616 0 5 2 0 9905 0 5 0 5 site2 Up 6 0 2 2 . . . elided . . . E4B2C5E4-0961-4F3A-A5F7-C3A4BB6B518C https://localhost:8089/services/cluster/manager/peers/E4B2C5E4-0961-4F3A-A5F7-C3A4BB6B518C 2014-04-17T19:17:08+00:00 system 4708B74780A1E5101449548B1E103616 0 4 13 6 7 ... elided ... 1 127.0.1.1:8094 1 s2p1 1397762227 4708B74780A1E5101449548B1E103616 0 7 2 0 9903 0 0 10 0 3 site2 Up 6 0 2 5cluster/manager/peers/{name}
https://:/services/cluster/manager/peers/{name}
Access specified peer.
GET
Get {name} peer information.
Request parameters
Name | Type | Description |
---|---|---|
list_buckets | Boolean | Indicates whether to list the buckets for the peers to this manager. |
Returned values
Name | Description |
---|---|
active_bundle_id | The ID of the configuration bundle currently being used by the manager. |
apply_bundle_status | Bundle status enumeration. |
base_generation_id | The initial bundle generation ID recognized by this peer. Any searches from previous generations fail.The initial bundle generation ID is created when a peer first comes online, restarts, or recontacts the manager. |
bucket_count | Count of the number of buckets on this peer, across all indexes. |
bucket_count_by_index | Count of the number of buckets by index on this peer. |
delayed_buckets_to_discard | List of bucket IDs waiting to be discarded on this peer. |
fixup_set | The set of buckets that need repair once you take the peer offline. |
heartbeat_started | Flag indicating if this peer has started heartbeating. |
host_port_pair | The host and port advertised to peers for the data replication channel.Can be either of the form IP:port or hostname:port. |
is_searchable | Flag indicating if this peer belongs to the current committed generation and is searchable. |
label | The name for the peer. Displayed on the Splunk Web manager page. |
last_heartbeat | Timestamp for last heartbeat recieved from the peer. |
latest_bundle_id | The ID of the configuration bundle this peer is using. |
pending_job_count | Used by the manager to keep track of pending jobs requested by the manager to this peer. |
primary_count | Number of buckets for which the peer is primary in its local site, or the number of buckets that return search results from same site as the peer. |
primary_count_remote | Number of buckets for which the peer is primary that are not in its local site. |
replication_count | Number of replications this peer is part of, as either source or target. |
replication_port | TCP port to listen for replicated data from another cluster member. |
replication_use_ssl | Indicates whether to use SSL when sending replication data. |
search_state_counter | Lists the number of buckets on the peer for each search state for the bucket.Possible values for search state include: Searchable Unsearchable |
site | To which site the peer belongs. |
splunk_version | The version of Splunk that the peer is running. This will be of the form X.Y.Z where X is the major version, Y is the minor version, and Z is the maintenance version. |
status | Indicates the status of the peer.Valid values are: Up Pending AutomaticDetention ManualDetention-PortsEnabled ManualDetention Restarting ShuttingDown ReassigningPrimaries Decommissioning GracefulShutdown Stopped Down BatchAdding |
status_counter | Lists the number of buckets on the peer for each bucket status.Possible values for bucket status: Complete: complete (warm/cold) bucket NonStreamingTarget: target of replication for already completed (warm/cold) bucket PendingTruncate: bucket pending truncation PendingDiscard: bucket pending discard Standalone: bucket that is not replicated StreamingError: copy of streaming bucket where some error was encountered StreamingSource: streaming hot bucket on source side StreamingTarget: streaming hot bucket copy on target side Unset: uninitialized |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/peers/29F9560E-A44A-425C-8753-1C6158B46C84
XML Response
clustermanagerpeers https://localhost:8089/services/cluster/manager/peers 2014-04-17T19🔞19+00:00 Splunk 1 30 0 29F9560E-A44A-425C-8753-1C6158B46C84 https://localhost:8089/services/cluster/manager/peers/29F9560E-A44A-425C-8753-1C6158B46C84 2014-04-17T19🔞19+00:00 system 4708B74780A1E5101449548B1E103616 0 3 11 6 5 ... elided ... ... elided ... 1 127.0.1.1:8092 1 s1p3 1397762298 4708B74780A1E5101449548B1E103616 0 6 2 0 9902 0 0 8 0 3 site1 7.2.0 Up 6 0 2 3cluster/manager/redundancy
https://:/services/cluster/manager/redundancy
Display the details of all cluster managers participating in cluster manager redundancy, and switch the HA state of the cluster managers.
Authentication and authorization
The GET on this endpoint needs the capability list_indexer_cluster
, and the POST on this endpoint needs the capability edit_indexer_cluster
.
GET
Display the details of all cluster managers participating in cluster manager redundancy.
Request parameters
None
Returned values
Name | Description |
---|---|
active_bundle_id | The active bundle ID of the cluster, as set in the given cluster manager. |
generation_id | The last committed generation ID of the cluster, as known to the given cluster manager. |
ha_mode | The high availability mode of the given cluster manager. |
last_heartbeat | The timestamp of the last heartbeat received from the given cluster manager. This is only applicable for the standby cluster managers. For the active cluster manager, this is set to 0. For standby cluster managers, this field reflects the valid timestamp, denoting the last time the active manager received a heartbeat from this standby cluster manager. |
manager_switchover_mode | The switchover mode set in the given cluster manager. |
peers_count | The number of indexer peers known to to the given cluster manager. |
server_name | The configured server name of the given cluster manager. |
uri | The management URI of the given cluster manager. |
Example request and response
Request
curl -k -u admin:changeme -XGET "https://mrt:15511/services/cluster/manager/redundancy/?output_mode=json"
Response
{ "links":{ "create":"/services/cluster/manager/redundancy/_new" }, "origin":"https://mrt:15511/services/cluster/manager/redundancy", "updated":"2022-01-25T08:29:41+00:00", "generator":{ "build":"e578ec650c0bf4d48e84541eae3d501f6dfc688a", "version":"20211229" }, "entry":[ { "name":"7EE219C0-23A6-4E95-A599-64E0FE5E8B05", "id":"https://mrt:15511/services/cluster/manager/redundancy/7EE219C0-23A6-4E95-A599-64E0FE5E8B05", "updated":"1970-01-01T00:00:00+00:00", "links":{ "alternate":"/services/cluster/manager/redundancy/7EE219C0-23A6-4E95-A599-64E0FE5E8B05", "list":"/services/cluster/manager/redundancy/7EE219C0-23A6-4E95-A599-64E0FE5E8B05", "edit":"/services/cluster/manager/redundancy/7EE219C0-23A6-4E95-A599-64E0FE5E8B05" }, "author":"system", "acl":{ "app":"", "can_list":true, "can_write":true, "modifiable":false, "owner":"system", "perms":{ "read":[ "admin", "splunk-system-role" ], "write":[ "admin", "splunk-system-role" ] }, "removable":false, "sharing":"system" }, "content":{ "active_bundle_id":"075EA8FB2D1172A1A7AD9DA472C63E92", "eai:acl":null, "generation_id":"21", "ha_mode":"Active", "last_heartbeat":0, "manager_switchover_mode":"auto", "peers_count":"5", "server_name":"cm", "uri":"https://mrt:15511" } }, { "name":"841BD315-21DB-4589-8813-15199DF02F1F", "id":"https://mrt:15511/services/cluster/manager/redundancy/841BD315-21DB-4589-8813-15199DF02F1F", "updated":"1970-01-01T00:00:00+00:00", "links":{ "alternate":"/services/cluster/manager/redundancy/841BD315-21DB-4589-8813-15199DF02F1F", "list":"/services/cluster/manager/redundancy/841BD315-21DB-4589-8813-15199DF02F1F", "edit":"/services/cluster/manager/redundancy/841BD315-21DB-4589-8813-15199DF02F1F" }, "author":"system", "acl":{ "app":"", "can_list":true, "can_write":true, "modifiable":false, "owner":"system", "perms":{ "read":[ "admin", "splunk-system-role" ], "write":[ "admin", "splunk-system-role" ] }, "removable":false, "sharing":"system" }, "content":{ "active_bundle_id":"075EA8FB2D1172A1A7AD9DA472C63E92", "eai:acl":null, "generation_id":"21", "ha_mode":"Standby", "last_heartbeat":1643099380, "manager_switchover_mode":"auto", "peers_count":"5", "server_name":"cm-standby2", "uri":"https://wimpy:14089" } } ], "paging":{ "total":2, "perPage":30, "offset":0 }, "messages":[
] }
POST
Switch the high availability state of the cluster managers.
Request parameters
_action=switch_mode
ha_mode=Active|Standby
Returned values
Name | Description |
---|---|
ha_mode | The resultant high availability mode of the given cluster manager after the mode change request completion. |
Example request and response
Request
curl -k -u admin:changeme -XPOST "https://10.16.88.2:15511/services/cluster/manager/redundancy/?output_mode=json" -d "_action=switch_mode" -d "ha_mode=Active"
Response
{ "links":{ "create":"/services/cluster/manager/redundancy/_new" }, "origin":"https://10.16.88.2:15511/services/cluster/manager/redundancy", "updated":"2021-10-14T04:15:00-07:00", "generator":{ "build":"42f3134682e376e692f6e407a83b41c8dd787e9e", "version":"20211011" }, "entry":[ { "name":"0AB9404D-8670-4F26-8723-CA289A5A0E3A", "id":"https://10.16.88.2:15511/services/cluster/manager/redundancy/0AB9404D-8670-4F26-8723-CA289A5A0E3A", "updated":"1969-12-31T16:00:00-08:00", "links":{ "alternate":"/services/cluster/manager/redundancy/0AB9404D-8670-4F26-8723-CA289A5A0E3A", "list":"/services/cluster/manager/redundancy/0AB9404D-8670-4F26-8723-CA289A5A0E3A", "edit":"/services/cluster/manager/redundancy/0AB9404D-8670-4F26-8723-CA289A5A0E3A" }, "author":"system", "acl":{ "app":"", "can_list":true, "can_write":true, "modifiable":false, "owner":"system", "perms":{ "read":[ "admin", "splunk-system-role" ], "write":[ "admin", "splunk-system-role" ] }, "removable":false, "sharing":"system" }, "content":{ "eai:acl":null, "ha_mode":"Active" } } ], "paging":{ "total":1, "perPage":30, "offset":0 }, "messages":[
] }
cluster/manager/sites
https://:/services/cluster/manager/sites
Access cluster site information.
GET
List available cluster sites.
Request parameters
None
Returned values
Name | Description |
---|---|
peers | Peers list of host:port and server name. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/sites
XML Response
clustermanagersites https://localhost:8089/services/cluster/manager/sites 2014-04-17T19:12:15+00:00 Splunk 2 30 0 site1 https://localhost:8089/services/cluster/manager/sites/site1 2014-04-17T19:12:15+00:00 system ... elided ... 127.0.1.1:8092 s1p3 127.0.1.1:8091 s1p2 127.0.1.1:8090 s1p1 site2 https://localhost:8089/services/cluster/manager/sites/site2 2014-04-17T19:12:15+00:00 system ... elided ... 127.0.1.1:8096 s2p3 127.0.1.1:8095 s2p2 127.0.1.1:8094 s2p1cluster/manager/sites/{name}
https://:/services/cluster/manager/sites/{name}
Access specific cluster site information.
GET
List the {name} cluster site information.
Request parameters
None
Returned values
Name | Description |
---|---|
peers | Site peer reference, for each peer. Possible values include the following.host_port_pair Peer port number. server_name Peer server name. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/sites/site1
XML Response
clustermanagersites https://localhost:8089/services/cluster/manager/sites 2014-04-17T19:13:07+00:00 Splunk 1 30 0 site1 https://localhost:8089/services/cluster/manager/sites/site1 2014-04-17T19:13:07+00:00 system ... elided ... ... elided ... 127.0.1.1:8092 s1p3 127.0.1.1:8091 s1p2 127.0.1.1:8090 s1p1cluster/manager/status
https://:/services/cluster/manager/status
Endpoint to get the status of a rolling restart.
GET
Get the status of a rolling restart.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
decommission_force_timeout | The amount of time, in seconds, the cluster manager will wait for a peer in primary decommission status to finish primary reassignmentand restart, during a searchable rolling restart with timeouts. Only valid for rolling_restart=searchable_force. Default value is 180. Max accepted value is 1800. |
maintenance_mode | Indicates if the cluster is in maintenance mode. Happens during rolling restart, bundle push, and other maintenance activities. |
messages | Array of messages from server. |
multisite | Indicates if multisite is enabled for this manager. Make sure you set site parameters on the peers if you set this to true. Defaults to false. |
peers | Object containing all the peers in the cluster. For each peer, the label, site and status are provided. |
restart_inactivity_timeout | The amount of time, in seconds, that the manager waits for a peer to restart and rejoin the cluster before it considers the restart a failure and proceeds to restart other peers. A value of zero (0) means that the manager waits indefinitely for a peer to restart. Only valid for rolling_restart=searchable_force. Default is 600secs. |
restart_progress | Object containing lists of peers in "done", "failed", "in_progress" and "to_be_restarted" state. |
rolling_restart_flag | Boolean that indicates if there is a rolling restart in progress. |
rolling_restart_or_upgrade | Boolean that indicates if there is a rolling restart or rolling upgrade in progress. |
searchable_rolling | Boolean that indicates if a searchable rolling restart/upgrade in progress. |
service_ready_flag | Boolean that indicates if the cluster is ready. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/manager/status
XML Response
clustermanagerstatus https://10.141.65.179:52000/services/cluster/manager/status 2018-04-01T23:00:53+00:00 Splunk 1 30 0 manager https://10.141.65.179:52000/services/cluster/manager/status/manager 1970-01-01T00:00:00+00:00 system 0 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system 0 0 idx3 default Up idx2 default Up idx1 default Up 0 0 0 0 1cluster/searchhead/generation
https://:/services/cluster/searchhead/generation
Access peer information in a cluster searchhead.
GET
List peers available to a cluster searchhead.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
generation_id | The current generation ID for this searchhead, which is part of a cluster configuration.The search head uses this information to determine which buckets to search across. |
generation_peers | List of peer nodes for the current generation in the cluster configuration for this searchhead. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/searchhead/generation
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clustersearchheadgeneration
https://localhost:8089/services/cluster/searchhead/generation
2012-09-05T11:13:45-07:00
Splunk
... opensearch nodes elided ...
manager
https://localhost:8089/services/cluster/searchhead/generation/manager
2012-09-05T11:13:45-07:00
system
... eai:acl node elided ...
2
splunks-ombra.sv.splunk.com:8389
splunks-ombra.sv.splunk.com
splunks-ombra.sv.splunk.com:8189
splunks-ombra.sv.splunk.com
cluster/searchhead/generation/{name}
https://:/services/cluster/searchhead/generation/{name}
Access peer of the manager URI.
GET
Get {name} searchhead generation ID and generation peers.
Request parameters
None
Returned values
Name | Description |
---|---|
generation_id | The current generation ID for this searchhead, which is part of a cluster configuration.The search head uses this information to determine which buckets to search across. |
generation_peers | List of peer nodes for the current generation in the cluster configuration for this searchhead. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/searchhead/generation/https%3A%252F%252Fmyserver-mbp15.sv.splunk.com%3A8989
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clustersearchheadgeneration
https://localhost:53791/services/cluster/searchhead/generation
2012-09-07T14:11:59-07:00
Splunk
1
30
0
https://ronnie.splunk.com:53112
https://localhost:53791/services/cluster/searchhead/generation/https%3A%252F%252Fronnie.splunk.com%3A53112
2012-09-07T14:11:59-07:00
system
... eai:acl node elided ...
... eai:attributes node elided ...
3
10.1.42.3:53309
peer3
10.1.42.3:53411
peer4
cluster/searchhead/searchheadconfig
https://:/services/cluster/searchhead/searchheadconfig
Access cluster searchhead node configuration.
GET
List this cluster search head node configuration.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/cluster/searchhead/searchheadconfig
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clustersearchheadconfig
https://localhost:8089/services/cluster/searchhead/searchheadconfig
2013-10-31T14:04:45-07:00
Splunk
... opensearch nodes elided ...
https://localhost:4567
https://myserver:7588/services/cluster/searchhead/searchheadconfig/https%3A%252F%252Flocalhost%3A4567
2013-10-31T14:04:45-07:00
system
... eai:acl node elided ...
https://localhost:4567
********
POST
Configure this server as a cluster searchhead node.
Request parameters
Name | Type | Description |
---|---|---|
name | String | Required. The URI of the manager node in the cluster. |
secret | String | Required. Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the manager, it is not able to communicate with the manager.Corresponds to pass4SymmKey setting in server.conf. |
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://myserver:8089/services/cluster/searchhead/searchheadconfig -d name=https://myserver:4567 -d secret=testsecret
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clustersearchheadconfig
https://localhost:8089/services/cluster/searchhead/searchheadconfig
2013-10-31T14:04:45-07:00
Splunk
... opensearch nodes elided ...
https://localhost:4567
https://myserver:8089/services/cluster/searchhead/searchheadconfig/https%3A%252F%252Flocalhost%3A4567
2013-10-31T14:04:45-07:00
system
... eai:acl node elided ...
https://localhost:4567
********
cluster/searchhead/searchheadconfig/{name}
https://:/services/cluster/searchhead/searchheadconfig/{name}
Manage node in a cluster.
DELETE
Remove node from cluster.
Request parameters
None
Returned values
None
Example request and response
XML Request
curl -k -u admin:changeme --request DELETE https://myserver.splunk.com:8089/services/cluster/searchhead/searchheadconfig/https%3A%252F%252Fmyserver%3A8211
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clustersearchheadconfig
https://myserver.splunk.com:8089/services/cluster/searchhead/searchheadconfig
2013-11-05T14:34:42-08:00
Splunk
0
30
0
GET
List cluster search head node configuration.
Request parameters
None
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://myserver.splunk.com:7588/services/cluster/searchhead/searchheadconfig/https%3A%252F%252Flocalhost%3A4567
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clustersearchheadconfig
https://myserver.splunk.com:8089/services/cluster/searchhead/searchheadconfig
2013-11-05T14:43:00-08:00
Splunk
... openserch nodes elided ...
https://localhost:4567
https://myserver.splunk.com:7588/services/cluster/searchhead/searchheadconfig/https%3A%252F%252Flocalhost%3A4567
2013-11-05T14:43:00-08:00
system
1
1
0
system
admin
splunk-system-role
admin
splunk-system-role
0
system
manager_uri
secret
https://localhost:4567
********
POST
Update cluster search head node configuration.
Request parameters
Name | Type | Description |
---|---|---|
manager_uri | String | The URI of the manager node in the cluster for which this searchhead is configured. |
secret | String | Secret shared among the nodes in the cluster to prevent any arbitrary node from connecting to the cluster. If a peer or searchhead is not configured with the same secret as the manager, it is not able to communicate with the manager.Corresponds to pass4SymmKey setting in server.conf. |
Returned values
None
cluster/peer/buckets
https://:/services/cluster/peer/buckets
Access cluster peers bucket configuration.
GET
List cluster peers bucket configuration.
Request parameters
Name | Type | Description |
---|---|---|
generation_id | String | The generation ID for this peer. For each generation, the manager server in a cluster configuration assigns generation IDs. A generation identifies which copies of a cluster's buckets are primary and therefore can participate in a search. |
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
checksum | Used internally to identify this bucket. |
earliest_time | Indicates the time of the earliest event in this bucket. |
generation_id | The generation ID for this peer. |
generations | A sparse list of generation id to bucket primacy for the given peer. |
latest_time | Indicates the time for the latest event in this bucket. |
search_state | Indicates if the bucket is searchable or unsearchable. |
status | Indicates the status of this bucket. One of the following values.Complete The copy of this bucket contains the full complement of information. StreamingSource The copy of this bucket is sending data to peer nodes for replication. StreamingTarget The copy of this bucket is receiving replicated data. NonStreamingTarget This copy of a warm bucket replication is in progress. Once replication is complete, the status changes to Complete. StreamingError The copy of this bucket encountered errors while streaming data. PendingTruncate The manager asked the peer to truncate this copy of the bucket to a certain size and is waiting for confirmation. PendingDiscard The manager asked the peer to discard this copy of the bucket (for whatever reason, and is waiting for confirmation. Standalone A bucket in the cluster that is not replicated. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/cluster/peer/buckets
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clusterpeerbuckets
https://localhost:8189/services/cluster/peer/buckets
2012-09-05T12:29:42-07:00
Splunk
... opensearch nodes elided ...
_audit~0~2AF11DD4-1424-4A14-A522-FB9D055E9516
https://localhost:8189/services/cluster/peer/buckets/_audit~0~2AF11DD4-1424-4A14-A522-FB9D055E9516
2012-09-05T12:29:42-07:00
system
... eai:acl node elided ...
1346859162
0x0
1346859257
Searchable
Complete
. . . elided ...
cluster/peer/buckets/{name}
https://:/services/cluster/peer/buckets/{name}
Manage peer buckets.
DELETE
Remove specified bucket from peer node.
Request parameters
Name | Type | Description |
---|---|---|
bucket_id | String | Required. The identifier for the bucket to remove. |
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass --request DELETE https://myserver:8089/services/cluster/peer/buckets/_internal~58~11111111-1111-1111-1111-111111111111 -d bucket_id="_internal5811111111-1111-1111-1111-111111111111"
XML Response
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:s="http://dev.splunk.com/ns/rest"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
clusterpeerbuckets
https://myserver:8089/services/cluster/peer/buckets
2013-10-31T14:48:18-07:00
Splunk
0
30
0
GET
List peer specified bucket information.
Request parameters
Name | Type | Description |
---|---|---|
generation_id | String | The generation ID for this peer. For each generation, the manager server in a cluster configuration assigns generation IDs. A generation identifies which copies of a cluster's buckets are primary and therefore can participate in a search. |
Returned values
Name | Description |
---|---|
checksum | Used internally to identify this bucket. |
earliest_time | Indicates the time of the earliest event in this bucket. |
generation_id | The generation ID for this peer. |
generations | A sparse list of generation id to bucket primacy for the given peer. |
latest_time | Indicates the time for the latest event in this bucket. |
search_state | Indicates if the bucket is Searchable or Unsearchable. |
status | Indicates the status of this bucket. One of the following values.Complete The copy of this bucket contains the full complement of information. StreamingSource The copy of this bucket is sending data to peer nodes for replication. StreamingTarget The copy of this bucket is receiving replicated data. NonStreamingTarget This copy of a warm bucket replication is in progress. Once replication is complete, the status changes to Complete. StreamingError The copy of this bucket encountered errors while streaming data. PendingTruncate The manager asked the peer to truncate this copy of the bucket to a certain size and is waiting for confirmation. PendingDiscard The manager asked the peer to discard this copy of the bucket (for whatever reason, and is waiting for confirmation. Standalone A bucket in the cluster that is not replicated. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/cluster/peer/buckets/_audit~0~B8B5E5C6-DB26-4952-AFB1-C5EFEFFFEA31
XML Response
. . . clusterpeerbuckets https://localhost:8189/services/cluster/peer/buckets 2012-09-05T12:40:43-07:00 Splunk ... opensearch nodes elided ... <s:messages/> _internal<del>1</del>50FCDB42-E167-458D-A6A9-E4587E8F16D9 https://localhost:8189/services/cluster/peer/buckets/_internal~1~50FCDB42-E167-458D-A6A9-E4587E8F16D9 2012-09-05T12:40:43-07:00 system <s:dict> <s:key name="checksum"> ... eai:acl node elided ... <s:key name="eai:attributes"> <s:dict> <s:key name="optionalFields"> <s:list/> <s:key name="requiredFields"> <s:list/> <s:key name="wildcardFields"> <s:list/> <s:key name="earliest_time">0 <s:key name="generations"> <s:dict> <s:key name="0">0xffffffffffffffff <s:key name="latest_time">0 <s:key name="search_state">Searchable <s:key name="status">StreamingSource
cluster/peer/control/control/decommission
https://:/services/cluster/peer/control/control/decommission
Endpoint to decommission an indexer cluster peer node.
POST
Decommission a peer node.
Returned values
None
Example request and response
XML Request
curl -k -u admin:pass https://indexer:8089/services/cluster/peer/control/control/decommission -X POST
XML Response
clusterpeercontrol https://10.141.66.19:46772/services/cluster/peer/control 2018-04-01T21:23:46+00:00 Splunk 0 30 0cluster/peer/control/control/re-add-peer
https://:/services/cluster/peer/control/control/re-add-peer
Set the peer to re-add itself to the manager. This syncs the peer's state, including its in-memory bucket state, to the manager. By default, this resets the peer's primary bucket copies and the manager reassigns them across the cluster. To keep the peer's existing primary bucket copies, use the optional clearMasks=false
parameter.
This endpoint can be useful when the manager and the peer have a state mismatch, for example when bucket information is not in sync between them.
POST
Re-add the cluster indexer to the cluster manager.
Request parameters
Name | Type | Default | Description |
---|---|---|---|
clearMasks | Boolean. Use true or false. | true | Optional. Indicates whether the manager should reassign all primary bucket copies across all peers. The default true value prompts the manager to reassign all primary bucket copies across all peers. Use false to re-add the peer but keep the existing primary bucket copies. |
Example request and response
XML Request
curl -k -u admin:changeme https://localhost:8089/services/cluster/peer/control/control/re-add-peer -d clearMasks=false -X POST
XML Response
clusterpeercontrol https://localhost:8089/services/cluster/peer/control 2015-11-06T18:08:54-08:00 Splunk 0 30 0cluster/peer/control/control/set_detention_override
https://:/services/cluster/peer/control/control/set_manual_detention
Deprecated. Use /set_manual_detention to manage peer node manual detention mode.
cluster/peer/control/control/set_manual_detention
https://:/services/cluster/peer/control/control/set_manual_detention
If you have Splunk Enterprise, you can use this endpoint to put the peer node in manual detention mode or take the peer out of this mode. In manual detention, the peer does not serve as a replication target. Detention helps slow the growth of disk usage on the peer.
Note:
- This endpoint replaces the
/set_detention_override
endpoint. - Starting with Splunk Enterprise software version 6.5, manual detention persists through restarts.
- For more information, see Put a peer in detention in Managing Indexers and Clusters of Indexers.
POST
Adjust cluster peer detention mode.
Request parameters
Name | Type | Description |
---|---|---|
manual_detention | Use one of the following values.off: Default. Remove the indexer from the detention state. on: Put the indexer in manual detention mode. Close the TCP, UDP, and HTTP Event Collector data ports. Closing the ports causes most external data indexing to stop during detention. on_ports_enabled: Put the indexer in manual detention mode. Do not close the TCP, UDP, or HTTP Event Collector data ports. The peer continues to index data during detention. | Enable or disable manual detention. Opt to close data ports or leave them open when manual detention is enabled. |
Returned values
None
Example request and response
XML Request
curl -k -u admin:password https://localhost:8089/services/cluster/peer/control/control/set_manual_detention -d manual_detention=on
XML Response
. . .
clusterpeercontrol https://localhost:8089/services/cluster/peer/control 2016-11-15T20:33:01-08:00 Splunk 0 30 0cluster/peer/info
https://:/services/cluster/peer/info
Access cluster peer node information.
GET
List peer information.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
active_bundle | Current bundle being used by this peer. |
base_generation_id | The initial bundle generation ID recognized by this peer. Any searches from previous generations fail.The initial bundle generation ID is created when a peer first comes online, restarts, or recontacts the manager. |
invalid_bundle_ids | List of bundle ids with validation errors in the peer. |
is_registered | Indicates if this peer is registered with the manager in the cluster. |
last_heartbeat_attempt | Timestamp for the last attempt to contact the manager. |
latest_bundle | Lists information about the most recent bundle downloaded from the manager. |
restart_state | Indicates whether the peer needs to be restarted to enable its cluster configuration. |
status | Indicates the status of the peer. One of the following values.Up Down Pending Detention Restarting DecommAwaitingPeer DecommFixingBuckets Decommissioned |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/cluster/peer/info
XML Response
. . . clusterpeerinfo https://localhost:8189/services/cluster/peer/info 2012-09-05T12:45:59-07:00 Splunk ... opensearch nodes elided ... <s:messages/> peer https://localhost:8189/services/cluster/peer/info/peer 2012-09-05T12:45:59-07:00 system <s:dict> <s:key name="active_bundle"> <s:dict> <s:key name="bundle_path">/Applications/splunk-peer/var/run/splunk/cluster/remote-bundle/0f6078895127ab1f715ee78a6e1ff8a1-1346858928.bundle <s:key name="checksum">36a883f4d47af66f78531ef474349b59 <s:key name="timestamp">1346858928 <s:key name="base_generation_id">2 ... eai:acl node elided ... <s:key name="invalid_bundle_ids"> <s:list/> <s:key name="is_registered">1 <s:key name="last_heartbeat_attempt">1346874358 <s:key name="latest_bundle"> <s:dict> <s:key name="bundle_path">/Applications/splunk-peer/var/run/splunk/cluster/remote-bundle/0f6078895127ab1f715ee78a6e1ff8a1-1346858928.bundle <s:key name="checksum">36a883f4d47af66f78531ef474349b59 <s:key name="timestamp">1346858928 <s:key name="restart_state">NoRestart <s:key name="status">Up
Search head cluster endpoints
The endpoints in this section pertain to search head clusters.
All endpoints contain shcluster
in their URIs pertain to search head clusters. For more information about search head clustering architecture, see Search head clustering architecture in the Distributed Search manual.
replication/configuration/health
https://:/services/replication/configuration/health
Access configuration replication health statistics for a search head cluster.
GET
Access the configuration replication health statistics for a search head cluster.
Request parameters
Name | Type | Description |
---|---|---|
bookmark | Boolean | Use this parameter with a GET request on the captain. Set to 1 to list the most recent changesets that members pulled from the captain. A timestamp is also returned for each changeset. |
check_share_baseline | Boolean | Set to 1 to check for a shared baseline among members. This parameter can be used with a request on any member, including the captain. |
unpublished | Boolean | Set to 1 to check for unpublished changes on members. Use this parameter with a request on a member to check if the member has any changes that have not been pushed to the captain. |
Returned values
Values returned depend on the request parameters used.
**bookmark**
Name | Description |
---|---|
[server_name] | For each [server_name] member, a changeset and timestamp are shown, indicating when the [server_name] member last pulled this set of configuration changes from the captain. |
Example request and response
curl -k -u admin:pass https://localhost:8089/services/replication/configuration/health?bookmark=1
... bookmark https://localhost:11089/services/replication/configuration/health/bookmark 2016-08-08T17:08:25-07:00 <s:dict> <s:key name="CaptainDummyOpId" title="undefined" rel="noopener noreferrer">https://localhost:11089">CaptainDummyOpId: Mon Aug 8 16:08:55 2016 <s:key name="2d9e86111eb4a377c60563f93ea5274de8b9c438" title="undefined" rel="noopener noreferrer">https://localhost:8089">2d9e86111eb4a377c60563f93ea5274de8b9c438: Mon Aug 8 17:08:22 2016 <s:key name="2d9e86111eb4a377c60563f93ea5274de8b9c438" title="undefined" rel="noopener noreferrer">https://localhost:9089">2d9e86111eb4a377c60563f93ea5274de8b9c438: Mon Aug 8 17:08:22 2016
**check_share_baseline**
Application usage
This parameter compares the baseline between the current instance, on which the GET request is made, with the baseline of other members. From each of the other members, the system retrieves the oldest changeset that is not more than 23 hours old and therefore safe from purging. The system then tries to find that changeset in the current instance's local changeset repository. If the changeset is found in the local repository, then the current instance and the member share a baseline.
Establishing a shared baseline between a captain and members is a prerequisite for successful configuration replication.
Name | Description |
---|---|
check_share_baseline | One of the following values is returned for each of the other members.Yes: The current instance shares a baseline with this node. No: The current instance node does not share a baseline with this node. Connection error: The current instance cannot contact this node. A warning is logged with additional details. |
server_name | Name for the member whose baseline is being compared to the current instance. |
Example request and response
curl -k -u admin:pass https://localhost:11089/services/replication/configuration/health?check_share_baseline=1
... health https://localhost:11089/services/replication/configuration/health 2016-08-09T15:51:06-07:00 Splunk <a href="https://localhost:8089" title="undefined" rel="noopener noreferrer">https://localhost:8089</a> https://localhost:11089/services/replication/configuration/health/https%3A%2F%2Flocalhost%3A8089 2016-08-09T15:51:06-07:00 <s:dict> <s:key name="check_share_baseline">Yes <s:key name="server_name">yxu-mbp15-node2 <a href="https://localhost:9089" title="undefined" rel="noopener noreferrer">https://localhost:9089</a> https://localhost:11089/services/replication/configuration/health/https%3A%2F%2Flocalhost%3A9089 2016-08-09T15:51:06-07:00 <s:dict> <s:key name="check_share_baseline">Yes <s:key name="server_name"> localhost-node3 <a href="https://localhost:11089" title="undefined" rel="noopener noreferrer">https://localhost:11089</a> https://localhost:11089/services/replication/configuration/health/https%3A%2F%2Flocalhost%3A11089 2016-08-09T15:51:06-07:00 <s:dict> <s:key name="check_share_baseline">Yes <s:key name="server_name"> localhost-node1 ...
**unpublished**
A Number of unpublished changes
key is returned with one of the following values.
Name | Description |
---|---|
0 | All changes on this cluster member have been pushed to the captain. There are no unpublished changes on this member. |
0 (This instance is the captain) | This message is returned when requesting unpublished status on the captain. The captain is always in sync with itself, so there are no unpublished changes. |
[Number greater than 0] | The number unpublished local changes on this member. Changes are held until the next replication occurs. The node is still healthy in this case. |
No captain is available | The search head cluster does not currently have a captain. |
Missing common baseline with the captain | This member might be out of sync with the captain if this message persists after several replication periods.This message can also appear during a transition period, for example, when a captain is switched or a member is manually resynced. On a healthy search head cluster, the unpublished value should return to a numeric value after one replication period. |
Example request and response
curl -k -u admin:pass https://localhost:11089/services/replication/configuration/health?unpublished=1
health https://localhost:8089/services/replication/configuration/health 2016-08-09T13:14:16-07:00 Splunk unpublished https://localhost:8089/services/replication/configuration/health/unpublished 2016-08-09T13:14:16-07:00 0replication/configuration/quarantined-assets
https://:/services/replication/configuration/quarantined-assets
Access information about quarantined lookups in a search head cluster.
GET
Access information about quarantined lookups in a search head cluster.
Request parameters
None
Returned values
Name | Description |
---|---|
assetName | The name of the quarantined CSV lookup. |
quarantined_at_host | The URL of the search head cluster member on which the lookup is quarantined. |
quarantined_at | Seconds since epoch. |
lookup_size | The size of the quarantined lookup in Bytes. |
Example request and response
curl -k -u admin:pass https://localhost:8090/services/replication/configuration/quarantined-assets
... quarantined-assets https://localhost:8090/services/replication/configuration/quarantined-assets/quarantined-assets 1970-01-01T00:00:00+00:00 <s:dict> <s:key name="assetId">b4c9340713a5dd8c61105b05acea79fbbd3fc98d <s:key name="assetURI">/nobody/search/lookups/test.csv <s:key name="user">nobody <s:key name="app">search <s:key name="assetType">lookups <s:key name="assetName">test.csv <s:key name="quarantineInfo">[ {quarantined_at_host=https://localhost:8090, quarantined_at=1724885036, lookup_size=30246329, quarantine_reason=large_lookup} ]
shcluster/captain/artifacts
https://:/services/shcluster/captain/artifacts
Provides list of artifacts and replicas currently managed by the captain across a searchhead cluster.
This endpoint can only be accessed on the captain. The response lists all artifacts that are currently resident on the set of search head cluster members.
An artifact in search head clustering is a managed search directory. Currently, only scheduled search results directories are managed and replicated according to replication policy.
Note: Ad hoc searches are not considered artifacts and are not listed.
GET
Lists searchhead cluster artifacts and replicas.
Request parameters
Name | Type | Description |
---|---|---|
remote_sids | Bool | Required. Set this to true to return the searches that the captain is seeing. Will include adhoc searches on remote members. |
Returned values
Name | Description |
---|---|
artifact_size | Artifact size, in bytes. |
origin_guid | Guid of the origin peer where this artifact was created/search was run. |
peers | Lists information about replicas of this artifact on members of this searchhead cluster. |
service_after_time | Artifact service/fixup is deferred until after this time. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/shcluster/captain/artifacts
XML Response
shclustercaptainartifacts https://localhost:8089/services/shcluster/captain/artifacts 2014-10-15T08:44:41-07:00 Splunk ... opensearch nodes elided ... scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413381600_268_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/captain/artifacts/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413381600_268_88888888-8888-8888-8888-888888888888 2014-10-15T08:44:41-07:00 system 282624 ... eai:acl node elided ... 88888888-8888-8888-8888-888888888888 /home/svasan/splunk/searchhead/var/run/splunk/dispatch/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413381600_268_88888888-8888-8888-8888-888888888888 Complete /home/svasan/splunk/dash/var/run/splunk/dispatch/rsa_scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413381600_268_88888888-8888-8888-8888-888888888888 Complete 0 scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413385200_281_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/captain/artifacts/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413385200_281_88888888-8888-8888-8888-888888888888 2014-10-15T08:44:41-07:00 system 282624 ... eai:acl node elided ... 88888888-8888-8888-8888-888888888888 /home/svasan/splunk/searchhead/var/run/splunk/dispatch/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413385200_281_88888888-8888-8888-8888-888888888888 Complete /home/svasan/splunk/dash/var/run/splunk/dispatch/rsa_scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413385200_281_88888888-8888-8888-8888-888888888888 Complete 0 scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387300_288_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/captain/artifacts/scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387300_288_88888888-8888-8888-8888-888888888888 2014-10-15T08:44:41-07:00 system 253952 ... eai:acl node elided ... 88888888-8888-8888-8888-888888888888 /home/svasan/splunk/searchhead/var/run/splunk/dispatch/scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387300_288_88888888-8888-8888-8888-888888888888 Complete /home/svasan/splunk/dash/var/run/splunk/dispatch/rsa_scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387300_288_88888888-8888-8888-8888-888888888888 Complete 0 scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387600_289_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/captain/artifacts/scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387600_289_88888888-8888-8888-8888-888888888888 2014-10-15T08:44:41-07:00 system 253952 ... eai:acl node elided ... 88888888-8888-8888-8888-888888888888 /home/svasan/splunk/searchhead/var/run/splunk/dispatch/scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387600_289_88888888-8888-8888-8888-888888888888 Complete /home/svasan/splunk/dash/var/run/splunk/dispatch/rsa_scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387600_289_88888888-8888-8888-8888-888888888888 Complete 0shcluster/captain/artifacts/{name}
https://:/services/shcluster/captain/artifacts/{name}
Get artifact information for a specific artifact.
GET
Get artifact information, size, replicas and earliest service time.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | ! Description |
---|---|
artifact_size | Artifact size, in bytes. |
origin_guid | Guid of the origin peer where this artifact was created. |
peers | Lists information about artifacts on members of this captain. |
service_after_time | Artifact service is deferred until after this time. |
Example request and response
XML Request
curl -k -u admin:pass https://locahost:8089/services/shcluster/captain/artifacts/scheduler__nobody__simplexml__RMD5dc07327042a35a17_at_1469214000_37_11111111-1111-1111-1111-111111111111
XML Response
shclustercaptainartifacts https://localhost:8089/services/shcluster/captain/artifacts 2016-07-22T13:39:03-07:00 Splunk 1 30 0 scheduler__nobody__simplexml__RMD5dc07327042a35a17_at_1469214000_37_11111111-1111-1111-1111-111111111111 https://localhost:8089/services/shcluster/captain/artifacts/scheduler__nobody__simplexml__RMD5dc07327042a35a17_at_1469214000_37_11111111-1111-1111-1111-111111111111 2016-07-22T13:39:03-07:00 system 77824 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system simplexml timechart_scheduled 11111111-1111-1111-1111-111111111111 /home/user/home_1/var/run/splunk/dispatch/scheduler__nobody__simplexml__RMD5dc07327042a35a17_at_1469214000_37_11111111-1111-1111-1111-111111111111 Complete /home/user/home_3/var/run/splunk/dispatch/rsa_scheduler__nobody__simplexml__RMD5dc07359042a35a17_at_1469214000_37_11111111-1111-1111-1111-111111111111 Complete read : [ *, splunk-system-user ], write : [ admin, power, splunk-system-user ] 0 splunk-system-usershcluster/captain/control/default/restart
https://:/services/shcluster/captain/control/default/restart
Endpoint to initiate rolling restart of a search head cluster.
POST
Initiates rolling restart of a search head cluster
Request parameters
Name | Type | Description |
---|---|---|
searchable | Boolean | Maintain high search availability during a rolling restart. |
force | Boolean | Override health check failures to continue searchable rolling restart. |
decommission_search_jobs_wait_secs | Integer | Maximum time in secs that searchable rolling restart waitsfor existing searches to finish. Default: 180 secs. |
Returned values
None
Example request and response
XML Request
curl -k -u admin:password https://localhost:8089/services/shcluster/captain/control/default/restart -d searchable=1 -d force=1 -d decommission_search_jobs_wait_secs=30 -X POST
XML Response
shclustercaptaincontrol https://10.222.21.58:8089/services/shcluster/captain/control 2018-03-29T12:08:09-07:00 Splunk 1 30 0 Restart of search head cluster members initiated. restart https://10.222.21.58:8089/services/shcluster/captain/control/restart 1969-12-31T16:00:00-08:00 system 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system Searchable rolling restarted cannot be started without captain status = Up, check status through "splunk show shcluster-status". 0shcluster/captain/control/control/rotate-splunk-secret
https://:/services/shcluster/captain/control/control/rotate-splunk-secret
Rotates the splunk.secret
file on all nodes of a search head cluster.
POST
Rotates the splunk.secret
file on all nodes of a search head cluster.
Returned values
None
Example request and response
XML Request
curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/control/control/rotate-splunk-secret -X POST
XML Response
shcluster/captain/control/control/upgrade-init
https://:/services/shcluster/captain/control/control/upgrade-init
Initializes a search head cluster rolling upgrade.
POST
Initializes a search head cluster rolling upgrade.
Returned values
None
Example request and response
XML Request
curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/control/control/upgrade-init -X POST
XML Response
shclustercaptaincontrol https://10.222.21.58:8089/services/shcluster/captain/control 2018-03-29T12:02:54-07:00 Splunk 1 30 0 Upgrade of search head cluster members initiated. upgrade-init https://10.222.21.58:8089/services/shcluster/captain/control/upgrade-init 1969-12-31T16:00:00-08:00 system 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system 1 yesshcluster/captain/control/control/upgrade-finalize
https://:/services/shcluster/captain/control/control/upgrade-finalize
Finishes a search head cluster rolling upgrade.
POST
Finishes a search head cluster rolling upgrade.
Returned values
None
Example request and response
XML Request
curl -k -u admin:changeme https://localhost:8089/services/shcluster/captain/control/control/upgrade-finalize -X POST
XML Response
shclustercaptaincontrol https://10.222.21.58:8089/services/shcluster/captain/control 2018-03-29T12:06:47-07:00 Splunk 1 30 0 Upgrade of search head cluster members finalized. upgrade-finalize https://10.222.21.58:8089/services/shcluster/captain/control/upgrade-finalize 1969-12-31T16:00:00-08:00 system 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system 1 noshcluster/captain/info
https://:/services/shcluster/captain/info
Access information about searchhead cluster captain node.
GET
List searchhead cluster captain node details.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
elected_captain | Time when the current captain was elected |
id | Id of this SH cluster. This is used as the unique identifier for the Search Head Cluster in bundle replication and acceleration summary management. |
initialized_flag | Indicates if the searchhead cluster is initialized. |
label | The name for the captain. Displayed on the Splunk Web manager page. |
maintenance_mode | Indicates if the cluster is in maintenance mode. |
min_peers_joined_flag | Flag to indicate if more then replication_factor peers have joined the cluster. |
peer_scheme_host_port | URI of the current captain. |
rolling_restart_flag | Indicates whether the captain is restarting the members in a searchhead cluster. |
service_ready_flag | Indicates whether the captain is ready to begin servicing, based on whether it is initialized. |
start_time | Timestamp corresponding to the creation of the captain. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/shcluster/captain/info
XML Response
shclustercaptaininfo https://localhost:8089/services/shcluster/captain/info 2014-10-15T08:45:25-07:00 Splunk ... opensearch nodes elided ... captain https://localhost:8089/services/shcluster/captain/info/captain 2014-10-15T08:45:25-07:00 system ... eai:acl node elided ... 1413307273 BB3116C0-73B9-459A-B473-254A18A69776 1 searchhead 0 1 https://localhost:55569 0 1 1413307203shcluster/captain/jobs
https://:/services/shcluster/captain/jobs
List running and recently finished jobs for all cluster members.
GET
List running and recently finished jobs for this cluster.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
For each job:
Name | Description |
---|---|
ATTEMPT_[n] | dispatch_time - The UTC time of dispatch for the job_errormsg_ - If the job failed, capturing the reason for failure_peer_ - GUID of the member that the job was sent to_sid_ - the search id of this attempt_success_ - a boolean for success/failure of the job |
job_state | Job State can be SCHEDULED/DISPATCHED/COMPLETED. A SCHEDULED job has been received by the captain from the scheduler to schedule. A DISPATCHED job has started to run on a remote member. A COMPLETED job has finished running on the remote member. |
saved_search | The name of the saved-search from the associated savedsearches.conf file. |
savedsearchtype | The scheduler manages three kinds of scheduled jobs, regular savedsearch for both realtime and historical, autosummary report acceleration build searches, and tsidx tsidx build searches. |
search_app | The application in which the savedsearch was created. |
search_owner | The owner of the saved search. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/shcluster/captain/jobs
XML Response
shclustercaptainjobs https://localhost:8089/services/shcluster/captain/jobs 2014-10-15T08:47:50-07:00 Splunk ... opensearch nodes elided ... savedsearch_Alert - syslog errors last hour_1087026166 https://localhost:8089/services/shcluster/captain/jobs/savedsearch_Alert%20-%20syslog%20errors%20last%20hour_1087026166 2014-10-15T08:47:50-07:00 system 1413363600 error response peer=https://wimpy.splunk.com:55560 rc=404 reason=' Application does not exist: SA-nix ' 99999999-9999-9999-9999-999999999999 NO_SID_RECEIVED_YET 0 1413363600 88888888-8888-8888-8888-888888888888 scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413363600_203_88888888-8888-8888-8888-888888888888 1 ... eai:acl node elided ... COMPLETED Alert - syslog errors last hour savedsearch SA-nix admin savedsearch_Alert - syslog errors last hour_11648853 https://localhost:8089/services/shcluster/captain/jobs/savedsearch_Alert%20-%20syslog%20errors%20last%20hour_11648853 2014-10-15T08:47:50-07:00 system 1413316800 error response peer=https://wimpy.splunk.com:55560 rc=404 reason=' Application does not exist: SA-nix ' 99999999-9999-9999-9999-999999999999 NO_SID_RECEIVED_YET 0 1413316800 88888888-8888-8888-8888-888888888888 scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413316800_34_88888888-8888-8888-8888-888888888888 1 ... eai:acl node elided ... COMPLETED Alert - syslog errors last hour savedsearch SA-nix admin . . . savedsearch_fired_alerts_1050236433 https://localhost:8089/services/shcluster/captain/jobs/savedsearch_fired_alerts_1050236433 2014-10-15T08:47:50-07:00 system 1413308100 error response peer=https://wimpy.splunk.com:55560 rc=404 reason=' Application does not exist: SA-nix ' 99999999-9999-9999-9999-999999999999 NO_SID_RECEIVED_YET 0 1413308100 88888888-8888-8888-8888-888888888888 scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413308100_2_88888888-8888-8888-8888-888888888888 1 ... eai:acl node elided ... COMPLETED fired_alerts savedsearch SA-nix adminshcluster/captain/jobs/{name}
https://:/services/shcluster/captain/jobs/{name}
GET
Get running and recently finished jobs for {name} cluster.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
ATTEMPT_[n] | dispatch_time - The UTC time of dispatch for the job_errormsg_ - If the job failed, capturing the reason for failure_peer_ - GUID of the member that the job was sent to_sid_ - the search id of this attempt_success_ - a boolean for success/failure of the job |
job_state | Job State can be SCHEDULED/DISPATCHED/COMPLETED. A SCHEDULED job has been received by the captain from the scheduler to schedule. A DISPATCHED job has started to run on a remote member. A COMPLETED job has finished running on the remote member. |
saved_search | The name of the saved-search from the associated savedsearches.conf file. |
savedsearchtype | The scheduler manages three kinds of scheduled jobs, regular savedsearch for both realtime and historical, autosummary report acceleration build searches, and tsidx tsidx build searches. |
search_app | The application in which the savedsearch was created. |
search_owner | The owner of the saved search. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/shcluster/captain/jobs/scheduled_sample%20scheduled%20search%20for%20dashboards%20%28existing%20job%20case%29%20timechart_12944444515
XML Response
shclustercaptainjobs https://localhost:8089/services/shcluster/captain/jobs 2016-07-22T13:56:18-07:00 Splunk 1 30 0 scheduled_sample scheduled search for dashboards (existing job case) timechart_1290934515 https://localhost:8089/services/shcluster/captain/jobs/scheduled_sample%20scheduled%20search%20for%20dashboards%20%28existing%20job%20case%29%20timechart_1290934515 2016-07-22T13:56:18-07:00 system 1469214120 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system COMPLETED 11111111-1111-1111-1111-111111111111 https://wimpy:13221 home-1 sample scheduled search for dashboards (existing job case) timechart scheduled testing nobody scheduler__nobody__testing__RMD5058c22ce2c07889b_at_1469214120_39_11111111-1111-1111-1111-111111111111 1shcluster/captain/members
https://:/services/shcluster/captain/members
Lists the search head cluster members.
GET
List cluster members.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
adhoc_searchhead | Flag to indicate if this member does not run scheduled searches. |
advertise_restart_required | Flag to indicate if this peer advertised that it needed a restart. |
artifact_count | Number of artifacts on this peer |
delayed_artifacts_to_discard | List of artifacts waiting to be deleted from this peer. |
fixup_set | N/A |
host_port_pair | The host and management port advertised by this peer. |
kv_store_host_port | Host and port of the kv store instance of this member. |
label | The name for this member. Displayed on the Splunk Web manager page. |
last_heartbeat | Timestamp for last heartbeat recieved from the peer |
peer_scheme_host_port | URI of the current captain. |
pending_job_count | Used by the captain to keep track of pending jobs requested by the captain to this member. |
replication_count | Number of replications this peer is part of, as either source or target. |
replication_port | TCP port to listen for replicated data from another cluster member. |
replication_use_ssl | Indicates whether to use SSL when sending replication data. |
site | N/A |
status | Indicates the status of the member. Possible values are the following.Up Pending AutomaticDetention ManualDetention-PortsEnabled ManualDetention Restarting ShuttingDown ReassigningPrimaries Decommissioning GracefulShutdown Stopped Down BatchAdding |
status_counter | Lists the number of buckets on the peer for each bucket status. Possible values are the following.Complete Complete (warm/cold) bucket NonStreamingTarget Target of replication for already completed (warm/cold) bucket PendingTruncate Bucket pending truncation PendingDiscard Bucket pending discard Standalone Bucket that is not replicated StreamingError Copy of streaming bucket where some error was encountered StreamingSource Streaming hot bucket on source side StreamingTarget Streaming hot bucket copy on target side Unset Uninitialized |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/shcluster/captain/members
XML Response
shclustercaptainmembers https://localhost:8089/services/shcluster/captain/members 2014-10-15T08:49:34-07:00 Splunk ... opensearch nodes elided ... 88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/captain/members/88888888-8888-8888-8888-888888888888 2014-10-15T08:49:34-07:00 system 0 0 4 ... eai:acl node elided ... localhost:8089 ? searchhead 1413388171 https://localhost:8089 0 0 3456 0 site2 Up 4 0 99999999-9999-9999-9999-999999999999 https://localhost:8089/services/shcluster/captain/members/99999999-9999-9999-9999-999999999999 2014-10-15T08:49:34-07:00 system 0 0 4 ... eai:acl node elided ... wimpy.splunk.com:55560 ? manager 1413388171 https://wimpy.splunk.com:55560 0 0 55570 0 site1 Up 4 0 0shcluster/captain/members/{name}
https://:/services/shcluster/captain/members/{name}
Get information about the {name} searchhead cluster member.
GET
Get information about the {name} searchhead cluster member.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
adhoc_searchhead | Flag to indicate if this member does not run scheduled searches. |
advertise_restart_required | Flag to indicate if this peer advertised that it needed a restart. |
artifact_count | Number of artifacts on this peer. |
delayed_artifacts_to_discard | List of artifacts waiting to be deleted from this peer. |
fixup_set | N/A |
host_port_pair | The host and management port advertised by this peer. |
kv_store_host_port | Host and port of the kv store instance of this member. |
label | The name for this member. Displayed on the Splunk Web manager page. |
last_heartbeat | Timestamp for last heartbeat recieved from the peer |
peer_scheme_host_port | URI of the current captain. |
pending_job_count | Used by the manager to keep track of pending jobs requested by the manager to this peer. |
replication_count | Number of replications this peer is part of, as either source or target. |
replication_port | TCP port to listen for replicated data from another cluster member. |
replication_use_ssl | Indicates whether to use SSL when sending replication data. |
site | N/A |
status | Indicates the status of the member.Possible values are the following. Up Pending AutomaticDetention ManualDetention-PortsEnabled ManualDetention Restarting ShuttingDown ReassigningPrimaries Decommissioning GracefulShutdown Stopped Down BatchAdding |
status_counter | Lists the number of buckets on the peer for each bucket status. Possible values are the following.Complete Complete (warm/cold) bucket NonStreamingTarget Target of replication for already completed (warm/cold) bucket PendingTruncate Bucket pending truncation PendingDiscard Bucket pending discard Standalone Bucket that is not replicated StreamingError Copy of streaming bucket where some error was encountered StreamingSource Streaming hot bucket on source side StreamingTarget Streaming hot bucket copy on target side Unset Uninitialized |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8089/services/shcluster/captain/members/33333333-3333-3333-3333-333333333333
XML Response
shclustercaptainmembers https://wimpy:13221/services/shcluster/captain/members 2016-07-22T14:12:50-07:00 Splunk 1 30 0 33333333-3333-3333-3333-333333333333 https://localhost:8089/services/shcluster/captain/members/33333333-3333-3333-3333-333333333333 2016-07-22T14:12:50-07:00 system 0 0 6 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system advertise_restart_required advertise_restart_required_reason alert_csv alert_csv_epoch artifacts_location_csv completed_summaries last_oaep last_osep partial_alert_delta partial_suppression_delta peer_load_stats_gla_15m peer_load_stats_gla_1m peer_load_stats_gla_5m peer_load_stats_max_runtime peer_load_stats_num_autosummary peer_load_stats_num_historical peer_load_stats_num_realtime peer_load_stats_num_running peer_load_stats_total_runtime peer_pid scheduler_disabled suppression_csv suppression_csv_epoch last_artifact_log_entry_processed last_si_entry_processed mgmt_port peer_load_stats queue_blocked_count wimpy:13223 0 wimpy:18323 wimpy-3 1469221966 https://wimpy:13223 0 https://wimpy:13223 0 1 0 12243 0 default Up 6 0 0shcluster/config
https://:/services/shcluster/config
List search head cluster node configuration.
GET
List search head cluster node configuration.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
cxn_timeout | Low-level timeout, in seconds, for establishing connection between searchhead cluster nodes. Defaults to 60 seconds. |
disabled | Indicates if this node is disabled. |
heartbeat_period | Only valid for member nodes in a searchhead cluster. The time, in seconds, that a member attempts to send a heartbeat to the captain |
heartbeat_timeout | Only valid for the captain node in a searchhead cluster configuration. The time, in seconds, before a captain considers a member down. Once a member is down, the captain initiates steps to replicate artifacts from the dead member to its live members. Defaults to 60 seconds. |
id | Id of the SH cluster this member is a part of. |
max_peer_rep_load | Maximum number of replications that can be ongoing as a target. |
mode | Valid values: (disabled, member, captain, dynamic_captain) Defaults to disabled. Multiple values are permitted.Sets operational mode for this searchhead cluster node. Only one captain may exist per searchhead cluster. |
percent_peers_to_restart | Percentage of peers to restart at the same time when doing a rolling restart. |
ping_flag | For internal use to facilitate communication between the captain and members. |
quiet_period | The time, in seconds, that a captain waits for members to add themselves to the searchhead cluster. |
rcv_timeout | Low-level timeout, in seconds, for receiving data between searchhead cluster nodes. Defaults to 60 seconds. |
register_replication_address | Valid only for nodes configured as members. The address on which a member is available for accepting replication data. This is useful in the cases where a member host machine has multiple interfaces and only one of them can be reached by another splunkd instance. |
rep_cxn_timeout | Low-level timeout, in seconds, for establishing a connection for replicating data. |
rep_max_rcv_timeout | Maximum cumulative time, in seconds, for receiving acknowledgement data from members. Defaults to 600s. |
rep_max_send_timeout | Maximum time, in seconds, for sending replication slice data between searchhead cluster nodes. Defaults to 600s. |
rep_rcv_timeout | Low-level timeout, in seconds, for receiving data between searchhead cluster nodes. |
rep_send_timeout | Low-level timeout, in seconds, for sending replication data between searchhead cluster nodes. Defaults to 5 seconds. |
replication_factor | Only valid for nodes configured as a captain.Determines how many copies of raw data are created in the searchhead cluster. This could be less than the number of searchhead cluster members.Must be greater than 0 and greater than or equal to the search factor. Defaults to 3. |
replication_port | TCP port to listen for replicated data from another searchhead cluster member. |
replication_use_ssl | Indicates whether to use SSL when sending replication data. |
restart_timeout | Only valid for nodes configured as a captain. The amount of time, in seconds, the captain waits for a member to come back when the member is restarted (to avoid the overhead of trying to fix the artifacts that were on the member). Defaults to 600 seconds.Note: This only works if the member is restarted from Splunk Web. |
secret | Secret shared among the nodes in the searchhead cluster to prevent any arbitrary node from connecting to the searchhead cluster. If a member or searchhead is not configured with the same secret as the captain, it is not able to communicate with the captain.Corresponds to pass4SymmKey setting in server.conf. |
send_timeout | Low-level timeout, in seconds, for sending data between searchhead cluster nodes. Defaults to 60 seconds. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/shcluster/config
XML Response
shclusterconfig https://localhost:8089/services/shcluster/config 2014-10-15T08:50:47-07:00 Splunk ... opensearch nodes elided ... config https://localhost:8089/services/shcluster/config/config 2014-10-15T08:50:47-07:00 system 60 0 ... eai:acl node elided ... 5 60 BB3116C0-73B9-459A-B473-254A18A69776 5 dynamic_captain 10 1 60 60 60 600 600 60 60 2 3456 0 60 ******** 60shcluster/config/config
https://:/services/shcluster/config/config
Configure search head cluster members.
POST
Configure search head cluster members.
Request parameters
Name | Type | Description |
---|---|---|
rolling_restart | String | Sets the mode for search head cluster rolling restart. Options include:restart: Initiates a rolling restart in classic mode (no guarantee of search continuity). searchable: Initiates a rolling restart with minimum search interruption. |
decommission_search_jobs_wait_secs | Integer | Specifies the amount of time, in seconds, that a search head cluster member waits for existing searches to complete before restarting. Default: 180 secs. |
manual_detention | Use one of the following values:off: Default. Remove the target search head from the detention state. on: Put the target search head in manual detention mode. | Specifies whether to put the cluster member in manual detention. |
target_uri | String | Specifies the target node you want to put in manual detention. |
Returned values
None
Example request and response
XML Request
curl -k -u admin:changed https://10.140.127.233:8089/services/shcluster/config/config -d rolling_restart=searchable -d decommission_search_jobs_wait_secs=120
XML Response
http://docs.splunk.com/Documentation/Splunk/7.2.0/RESTREF/RESTcluster
Example request and response for manual detention of a cluster member
XML Request
curl -k -u admin:changedpwd https://fool01.sv.splunk.com:8095/services/shcluster/config/config -d manual_detention=on -d target_uri=https://test.sv.splunk.com:8080
XML Response
shclusterconfig https://10.140.127.233:8089/services/shcluster/config 2018-04-02T16:16:08-07:00 Splunk 0 30 0shcluster/member/artifacts
https://:/services/shcluster/member/artifacts
Manage searchhead cluster member artifact configuration.
GET
List searchhead cluster members artifact configuration.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
For each member:
Name | Description |
---|---|
status | Indicates the status of this artifact. Possible values are as follows.Complete The copy of this artifact contains the full complement of information. StreamingSource The copy of this artifact is sending data to member nodes for replication. StreamingTarget The copy of this artifact is receiving replicated data. NonStreamingTarget This copy of a warm artifact replication is in progress. Once replication is complete, the status changes to Complete. StreamingError The copy of this artifact encountered errors while streaming data. PendingTruncate The captain asked the member to truncate this copy of the artifact to a certain size and is waiting for confirmation. PendingDiscard The captain asked the member to discard this copy of the artifact and is waiting for confirmation. Standalone An artifact in the searchhead cluster that is not replicated. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/shcluster/member/artifacts
XML Response
shclustermemberartifacts https://localhost:8089/services/shcluster/member/artifacts 2014-10-15T08:51:46-07:00 Splunk ... opensearch nodes elided ... scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413381600_268_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/member/artifacts/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413381600_268_88888888-8888-8888-8888-888888888888 2014-10-15T08:51:46-07:00 system ... eai:acl node elided ... Complete scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413385200_281_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/member/artifacts/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413385200_281_88888888-8888-8888-8888-888888888888 2014-10-15T08:51:46-07:00 system ... eai:acl node elided ... Complete scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387900_290_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/member/artifacts/scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413387900_290_88888888-8888-8888-8888-888888888888 2014-10-15T08:51:46-07:00 system ... eai:acl node elided ... Complete scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413388200_291_88888888-8888-8888-8888-888888888888 https://localhost:8089/services/shcluster/member/artifacts/scheduler__admin_U0Etbml4__RMD5b9b800e209365567_at_1413388200_291_88888888-8888-8888-8888-888888888888 2014-10-15T08:51:46-07:00 system ... eai:acl node elided ... Completeshcluster/member/artifacts/{name}
https://:/services/shcluster/member/artifacts/{name}
Get {name} member artifact configuration.
GET
List {name} member artifact information.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
Name | Description |
---|---|
status | Indicates the status of this artifact. Possible values are as follows.Complete The copy of this artifact contains the full complement of information. StreamingSource The copy of this artifact is sending data to member nodes for replication. StreamingTarget The copy of this artifact is receiving replicated data. NonStreamingTarget This copy of a warm artifact replication is in progress. Once replication is complete, the status changes to Complete. StreamingError The copy of this artifact encountered errors while streaming data. PendingTruncate The captain asked the member to truncate this copy of the artifact to a certain size and is waiting for confirmation. PendingDiscard The captain asked the member to discard this copy of the artifact and is waiting for confirmation. Standalone An artifact in the searchhead cluster that is not replicated. |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/shcluster/member/artifacts/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413518400_762_88888888-8888-8888-8888-888888888888
XML Response
shclustermemberartifacts https://localhost:8089/services/shcluster/member/artifacts 2014-10-16T22:33:37-07:00 Splunk ... opensearch nodes elided ... scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413518400_762_88888888-8888-8888-8888-888888888888 https://wimpy.splunk.com:55569/services/shcluster/member/artifacts/scheduler__admin_U0Etbml4__RMD592d31e53ed62579e_at_1413518400_762_88888888-8888-8888-8888-888888888888 2014-10-16T22:33:37-07:00 system ... eai:acl node elided ... ... eai:attributes node elided ... Completeshcluster/member/control/control/set_manual_detention
https://://services/shcluster/member/control/control/set_manual_detention
Put the search head cluster member in manual detention mode or take the search head cluster member out of this mode. When a search head cluster member is in manual detention, it does not accept new search jobs, including both scheduled and ad-hoc searches. Existing search jobs run to completion. It also participates in cluster administration operations with the exception of artifact replication.
POST
Adjust search head manual detention mode.
Request parameters
Name | Type | Description |
---|---|---|
manual_detention | Use one of the following values.off: Default. Remove the search head from the detention state. on: Put the search head in manual detention mode. | Enable or disable manual detention. |
Returned values
None
Example request and response
XML Request
curl -u admin:password -k https://localhost:8089/servicesNS/admin/search/shcluster/member/control/control/set_manual_detention -d manual_detention=on
XML Response
shclustermembercontrol https://localhost:8089/servicesNS/admin/search/shcluster/member/control 2018-03-28T08:04:28-07:00 Splunk 0 30 0shcluster/member/consensus
https://:/services/shcluster/member/consensus
Get latest cluster configuration from the raft consensus protocol.
GET
Get latest cluster configuration from the raft consensus protocol.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
These values are returned for each member.
Name | Description |
---|---|
configuration_id | Unique id for this configuration. |
servers_list | Comma-separated list of members that are part of the cluster. Each member is listed as scheme://host:port |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/shcluster/member/consensus
XML Response
shclustermemberconsensus https://localhost:8089/services/shcluster/member/consensus 2014-10-15T08:52:28-07:00 Splunk ... opensearch nodes elided ... shc_cluster_configuration https://localhost:8089/services/shcluster/member/consensus/shc_cluster_configuration 2014-10-15T08:52:28-07:00 system 4 ... eai:acl node elided ... https://localhost:55560,https://localhost:55569shcluster/member/info
https://:/services/shcluster/member/info
Access searchhead cluster member node information.
GET
List member information.
Request parameters
Pagination and filtering parameters can be used with this method.
Returned values
These values are returned for each member.
Name | Description |
---|---|
active_historical_search_count | Number of currently running historical searches. |
active_realtime_search_count | Number of currently running realtime searches. |
adhoc_searchhead | Flag that indicates if this member can run scheduled searches. |
is_registered | Indicates if this member is registered with the searchhead cluster captain. |
last_heartbeat_attempt | Timestamp for the last attempt to contact the captain. |
maintenance_mode | N/A |
peer_load_stats_gla_15m | Number of scheduled searches run in the last 15 minutes. |
peer_load_stats_gla_1m | Number of scheduled searches run in the last one minute. |
peer_load_stats_gla_5m | Number of scheduled searches run in the last five minutes. |
peer_load_stats_max_runtime | N/A |
peer_load_stats_num_autosummary | N/A |
peer_load_stats_num_historical | N/A |
peer_load_stats_num_realtime | N/A |
peer_load_stats_num_running | N/A |
peer_load_stats_total_runtime | N/A |
restart_state | Indicates whether the member needs to be restarted to enable its searchhead cluster configuration. |
status | Indicates the status of the member. Possible values are as follows.Up Pending AutomaticDetention ManualDetention Restarting ShuttingDown ReassigningPrimaries Decommissioning GracefulShutdown Down |
Example request and response
XML Request
curl -k -u admin:pass https://localhost:8189/services/shcluster/member/info
XML Response
shclustermemberinfo https://10.222.21.58:8089/services/shcluster/member/info 2018-03-29T12:05:35-07:00 Splunk 1 30 0 member https://10.222.21.58:8089/services/shcluster/member/info/member 1969-12-31T16:00:00-08:00 system 0 0 0 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system 1 1522350335 0 0 0 0 0 0 0 0 0 0 0 NoRestart ManualDetentionshcluster/status
https://:/services/shcluster/status
Performs health checks to determine search head cluster health status, prior to a rolling upgrade or rolling restart.
Authentication and Authorization
Requires the admin
role or list_search_head_clustering
capability.
GET
Get search head cluster health status information .
Request parameters
Name | Type | Description |
---|---|---|
advanced | Boolean | Lists search head cluster status information in a verbose manner. |
Returned values
These values are returned for each member.
Node | Name | Datatype | Description |
---|---|---|---|
Captain | decommission_search_jobs_wait_secs | Integer | Determines the maximum time, in seconds, that a member waits for search jobs to complete before it transitions to the down or GracefulShutdown state. |
dynamic_captain | Boolean | If true (1), then the captain is selected by elections. If false (0), then a static captain (no elections) is assigned. | |
elected_captain | String | The time when new captain is elected. | |
id | String | Specifies the search head cluster GUID. | |
initialized_flag | Boolean | Indicates if the captain is initialized. | |
label | String | Specifies the search head cluster label. | |
max_failures_to_keep_majority | Boolean | Indicates how many more nodes can be down to keep majority. | |
mgmt_uri | String | Specifies the URI and management port for the captain. | |
min_peers_joined_flag | Boolean | min_peers_joined_flag is true when there are at least as many search head peers as the replication_factor. | |
rolling_restart | String | Shows the restart mode, either restart or searchable. | |
rolling_restart_flag | Boolean | rolling_restart_flag is true when a rolling restart is in progress. | |
rolling_upgrade_flag | Boolean | rolling_upgrade_flag is true when a rolling upgrade is in progress. | |
service_ready_flag | Boolean | service_ready_flag is true when everything is up and running as expected and "ready to go!" | |
stable_captain | Boolean | Indicates stable captain based on heartbeat. | |
Member | label | String | Specifies the search head label. |
last_conf_replication | String | Specifies when the member last pulled a set of configurations from the captain. | |
manual_detention | String | Indicates if the member is in manual detention. Use off or on. | |
mgmt_uri | String | Specifies the URI and management port for the member. | |
mgmt_uri_alias | String | Specifies the URI and management port for the member. | |
out_of_sync_node | Boolean | out_of_sync_node is true when the member is out of sync. | |
preferred_captain | Boolean | Indicates the member's preference for captaincy. | |
restart_required | Boolean | restart_required is true when member requests a restart | |
splunk_version | String | Splunk version running on the search head. | |
status | String | Indicates the current status of the member. |
Example request and response
XML Request
curl -k -u admin:changed123 https://localhost:8089/services/shcluster/status?advanced=1?
XML Response
shclusterstatus https://10.222.21.58:8089/services/shcluster/status 2018-03-29T12:00:50-07:00 Splunk 1 30 0 status https://10.222.21.58:8089/services/shcluster/status/status 1969-12-31T16:00:00-08:00 system 1 Thu Mar 29 11:58:04 2018 93E0DBE8-A435-462F-BF7D-6297C9D9F939 1 ip-10-222-21-58 https://10.222.21.58:8089 1 0 1 1 1 0 system admin splunk-system-role admin splunk-system-role 0 system ip-10-222-25-57 Thu Mar 29 12:00:49 2018 https://10.222.25.57:8089 https://10.222.25.57:8089 Up ip-10-222-31-70 Thu Mar 29 12:00:46 2018 https://10.222.31.70:8089 https://10.222.31.70:8089 Up ip-10-222-21-58 https://10.222.21.58:8089 https://10.222.21.58:8089 Upupgrade/shc/recovery
https://:/services/upgrade/shc/recovery
Return search head cluster to ready state after automated rolling upgrade failure.
Authentication and Authorization
Requires admin role or another role containing these capabilties:
- upgrade_splunk_shc
- list_search_head_clustering
- list_settings
- use_remote_proxy
POST
Return SHC to ready state after automated rolling upgrade failure.
Request parameters
None
Returned values
Name | Type | Description |
---|---|---|
status | String | Status of HTTP request. For example, "succeeded" or "failed" |
Example request and response
JSON Request
curl -X POST -u admin:pass -k https://localhost:8089/services/upgrade/shc/recovery?output_mode=json
JSON Response
{ "updated":"2022-11-24T17:36:20+0000", "author":"Splunk", "layout":"props", "entry":[ { "title":"recovery", "id":"/services/upgrade/shc/recovery", "updated":"2022-11-24T17:36:20+0000", "links":{ "alternate":{ "href":"shc/recovery" } }, "content":{ "message":"Instance recovered successfully", "status":"succeeded" } } ] }
upgrade/shc/status
https://:/services/upgrade/shc/status
Check the status of an automated search head cluster rolling upgrade.
Authentication and Authorization
Requires admin role or another role containing these capabilities:
- upgrade_splunk_shc
- list_search_head_clustering
- list_settings
- use_remote_proxy
GET
Check the status of automated SHC rolling upgrade.
Request parameters
None
Returned values
Name | Type | Description |
---|---|---|
upgrade status | String | Status of automated rollling upgrade for entire clutser. |
peers_to_upgrade | Number | The total number of cluster members to upgrade. |
overall_peers_upgraded | Number | The number of cluster members upgraded at present. |
overall_peers_upgraded_percentage | Number | The percentage of total cluster members upgraded at present. |
name | String | The name of the individual cluster member. |
status | String | Upgrade status of the individual cluster member. |
last_modified | String | Date and time the individual cluster member was modified. |
Example request and response
JSON Request
curl -u admin:pass -k https://localhost:8089/services/upgrade/shc/status?output_mode=json
JSON Response
{ "updated":"2022-11-24T17:33:28+0000", "author":"Splunk", "layout":"props", "entry":[ { "title":"status", "id":"/services/upgrade/shc/status", "updated":"2022-11-24T17:33:28+0000", "links":{ "alternate":{ "href":"shc/status" } }, "content":{ "message":{ "upgrade_status":"completed", "statistics":{ "peers_to_upgrade":3, "overall_peers_upgraded":3, "overall_peers_upgraded_percentage":100 }, "peers":[ { "name":"sh2", "status":"upgraded", "last_modified":"Thu Nov 24 17:29:41 2022" }, { "name":"sh1", "status":"upgraded", "last_modified":"Thu Nov 24 17:28:07 2022" }, { "name":"sh3", "status":"upgraded", "last_modified":"Thu Nov 24 17:31:15 2022" } ] } } } ] }
upgrade/shc/upgrade
https://:/services/upgrade/shc/upgrade
Initiate an automated rolling upgrade of a search head cluster.
Authentication and Authorization
Requires admin role or another role containing these capabilities:
- upgrade_splunk_shc
- list_search_head_clustering
- list_settings
- use_remote_proxy
POST
Initiate automated SHC rolling upgrade.
Request parameters
None
Returned values
Name | Type | Description |
---|---|---|
status | String | Status of HTTP request. For example, "succeeded" or "failed". |
Example request and response
JSON Request
curl -X POST -u admin:pass -k https://localhost:8089/services/upgrade/shc/upgrade?output_mode=json
JSON Response
{ "updated":"2022-11-24T17:25:54+0000", "author":"Splunk", "layout":"props", "entry":[ { "title":"upgrade", "id":"/services/upgrade/shc/upgrade", "updated":"2022-11-24T17:25:54+0000", "links":{ "alternate":{ "href":"shc/upgrade" } }, "content":{ "message":"Upgrade initiated", "status":"succeeded" } } ] }