Splunk remote deployments (splunkremotesearch)
Overview of Splunk Remote Search capabilities in TrackMe
TrackMe can manage locally available data, as well as data from remote Splunk deployments (Splunk Enterprise, Splunk Cloud) and remote independent instances such as utility nodes (Deployment Servers, Cluster managers, License Manager, etc) or Heavy Forwarders.
This is a key capability provided by TrackMe, which is used transparently when you configure trackers, allowing you to target either the local environment where TrackMe is hosted, or any other Splunk remote deployment.
There are plenty of use cases where this capability of TrackMe is game changing, enabling TrackMe to become the single pane of glass for your monitoring and operations.
![diagram.png](_images/diagram1.png)
The Splunk remote search feature in TrackMe relies on multiple aspects:
The
splunkremotesearch
TrackMe generating custom command, which relies on the Splunk Python SDK to perform searches remotely at scaleThe concept of accounts which allows TrackMe to store and access pre-configured configuration defining the Splunk remote environment
Various backend level features in TrackMe designed to identify and handle the remote entities accordingly
The Remote deployment concept can be used in for various purposes, from feeds tracking on remote deployments, to Flex objects, CIM compliance tracking or Workload between different Search Head tiers
Note
local service account user or SAML service account
On the remote Search Heads tier counterpart, you can use a local user or a SAML user associated with the bearer token
However, to use SAML, your SAML setup needs to support AQR and it needs to be configured
Reference: https://docs.splunk.com/Documentation/Splunk/latest/Security/Setupauthenticationwithtokens
Reference: https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/SAMLConfigJWT
Hint
TrackMe supports multiple REST endpoints per account, with High Availability and Disaster Recovery capabilities: (from version 2.0.22)
For each account, you can specify a comma separated list of REST endpoints per account
TrackMe will automatically verifies the connectivity and randomly choose a REST endpoint target between any reachable endpoints
Therefore, you can specify multiple endpoints to support high availability and distribution of the searches against multiple endpoints automatically
This capability can useful for example if you deal with a Search Head Cluster and cannot use a load balancer or VIP to access to the cluster
Hint
TrackMe supports and requires Roles Based Access Control per Remote account defined: (from version 2.0.34)
For each account, you need to specify a comma separated list of Splunk roles that are allowed to use this Remote account
For retro-compatibility purposes, if the roles for an existing account have not been setup yet, TrackMe will use builtin roles in addition with typical admin roles (admin, sc_admin, trackme_user, trackme_power, trackme_admin)
If the user calling the
splunkremotesearch
command is not a member of the account specified roles, access will be refusedFrom version 2.0.61, both direct membership and inheritance are supported, users need to be a member any of the provided roles, or a role which inherites from any of these roles
Minimal RBAC requirements for the user account
TrackMe remote capabilities rely on a Splunk bearer token authentication, this token is associated with a Splunk user on the remote side which itself is associated with specific roles, capabilities, permissions and resources restrictions:
Roles and capabilities: The user can be created with minimal permissions using the Splunk
user
role out of the box role. (you can inherit from user or a role providing the same capabilities than power)Indexes: Make sure the user can access to both normal and
internal
indexes.Restrictions: The user for TrackMe should not have any time limits restrictions, there are use cases which require long term searches.
Resources: It is recommended to give to this user
enough concurrent searches
(unlike very basic or minimal user) as well as asufficient quota
. (5GB or 10Gb for instance)
![screen01.png](_images/screen012.png)
![screen03.png](_images/screen032.png)
![screen04.png](_images/screen041.png)
![screen05.png](_images/screen05.png)
Configuring a new remote account
When configuring a Splunk Remote deployment, you will provide:
A name which uniquely identify the account (the account ID)
The URL of the Splunk Search Head REST API, this can be Fully Qualified Domain Name, a hostname, an IP address (which itself can be a load balancer)
Multiple REST endpoints can be submitted as a comma separated list of targets
The bearer token value which is the secured credentials to access the environment, the bearer token is securely stored in the Splunk credential store
The application name space where remote searches are to be executed, defaults to the Search application on the remote target
A comma separated list of Splunk roles allowed to use this account (membership or inheritance)
At the time of the creation or later on if you edit the account, a connectivity check is automatically performed which validates the network connectivity and authentication, if the check fails the interface will refuse the requested action
To configure a new remote account, access the Configuration user interface from the navigation bar:
![screen1.png](_images/screen125.png)
If the connectivity check fails at the creation step, the configuration UI will raise an exception which indicates the root cause of the failure:
![screen_fail.png](_images/screen_fail.png)
Once the account exists, you can test the connectivity easily using the following TrackMe REST endpoint in SPL:
| trackme mode=post url="/services/trackme/v2/configuration/test_remote_account" body="{'account': '<name of the account>'}"
![screen2.png](_images/screen220.png)
The endpoint verifies that TrackMe can connect and authenticate successfully to the Splunk remote deployment.
When creating a new Virtual Tenant or a new Hybrid object, TrackMe will perform the same connectivity verification automatically:
![screen3.png](_images/screen38.png)
Example of a Splunk Remote search performed by TrackMe
When performing Remote Splunk searches, TrackMe automatically defines an optimised search to split the search logic between the part executed remotely, and the search logic handled locally.
Example:
![screen4.png](_images/screen47.png)
Using splunkremotesearch in a different application namespace
To use the command splunkremotesearch
from a different application than TrackMe, such as Search & Reporting, you need to perform a few steps as follows.
Manage capabilities
TrackMe requires capabilities to be able to access the command splunkremotesearch
, the user roles need to have the capability:
trackmeuseroperations
You can choose to inherit from the trackme_user
role, or you can add the capability to your user roles as needed.
Note: Access is granted by any of the 3 builtin trackme_ user roles*
see: Role Based Access Control and ownership for more information about capabilities in TrackMe.
Manage Roles Based Access control on the account
User roles needs to be listed in the account, by default when creating an account, the following roles are listed:
admin, sc_admin, trackme_user, trackme_power, trackme_admin
This is an explicit membership requirement, in our example we grant the user role “soc” to allow a member of this role to use the splunkremotesearch
command against this account:
![share002.png](_images/share002.png)
Use the command
In our example, the “soc” user is a member of a role with the same name, and can now use splunkremotesearch
command as needed to perform searches against the remote deployment, from any other application namespace:
![share003.png](_images/share003.png)
Accesses on the remote deployment rely on the service account which is associated with the bearer token configured in the account in TrackMe.
Troubleshooting failure to create or update a remote account
TrackMe Remote Search capabilities are an API based integration with Splunk API relying on Bearrer token authentication, you can refer to:
When attempting to configure a new remote account, or update an existing account, TrackMe will attempt to validate the connectivity, both from a network and authentication perspective.
If this step fails, the UI will clearly raise and show the exception encountered, for instance:
Example of a network connectivity failure: (wrong host)
![share003.png](_images/troubleshoot_netissue.png)
Example of a network connectivity failure: (timeout)
![share003.png](_images/troubleshoot_timeout.png)
Example of a refused authentication by the remote counter part:
![share003.png](_images/troubleshoot_refused.png)
Note: You can also test the connectivity in pure SPL via the following TrackMe REST API endpoint:
| trackme mode=post url="/services/trackme/v2/configuration/test_remote_connectivity" body="{'target_endpoints': 'https://mysplunk.mydomain.com:8089', 'bearer_token': 'xxxx', 'app_namespace': 'search'}"
Inspect splunkd.log logs on the remote counter part:
If the connectivity fails for authentication related reasons, such as a wrong token or lack of capabilities, verify the splunkd.log on the remote counter part, you can use the following Splunk search:
index=_internal source=*/splunkd.log JsonWebToken
Messages will be logged by Splunk, eventually indicating the root cause of the failure.
Example of a refused token:
02-05-2024 17:30:44.226 +0000 ERROR JsonWebToken [2843158 TcpChannelThread] - JsonWebToken validation failed because: Token xxxx-xxxx has an invalid signature.