Splunk remote deployments (splunkremotesearch)

Overview of Splunk Remote Search capabilities in TrackMe

TrackMe can manage locally available data, as well as data from remote Splunk deployments (Splunk Enterprise, Splunk Cloud) and remote independent instances such as utility nodes (Deployment Servers, Cluster managers, License Manager, etc) or Heavy Forwarders.

This is a key capability provided by TrackMe, which is used transparently when you configure trackers, allowing you to target either the local environment where TrackMe is hosted, or any other Splunk remote deployment.

There are plenty of use cases where this capability of TrackMe is game changing, enabling TrackMe to become the single pane of glass for your monitoring and operations.

The Splunk remote search feature in TrackMe relies on multiple aspects:

• The splunkremotesearch TrackMe generating custom command, which relies on the Splunk Python SDK to perform searches remotely at scale

• The concept of accounts which allows TrackMe to store and access pre-configured configuration defining the Splunk remote environment

• Various backend level features in TrackMe designed to identify and handle the remote entities accordingly

• The Remote deployment concept can be used in for various purposes, from feeds tracking on remote deployments, to Flex objects, CIM compliance tracking or Workload between different Search Head tiers

Note

local service account user or SAML service account

• On the remote Search Heads tier counterpart, you can use a local user or a SAML user associated with the bearer token

• However, to use SAML, your SAML setup needs to support AQR and it needs to be configured

• Reference: https://docs.splunk.com/Documentation/Splunk/latest/Security/Setupauthenticationwithtokens

• Reference: https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/SAMLConfigJWT

Hint

TrackMe supports multiple REST endpoints per account, with High Availability and Disaster Recovery capabilities: (from version 2.0.22)

• For each account, you can specify a comma separated list of REST endpoints per account

• TrackMe will automatically verifies the connectivity and randomly choose a REST endpoint target between any reachable endpoints

• Therefore, you can specify multiple endpoints to support high availability and distribution of the searches against multiple endpoints automatically

• This capability can useful for example if you deal with a Search Head Cluster and cannot use a load balancer or VIP to access to the cluster

Hint

TrackMe supports and requires Roles Based Access Control per Remote account defined: (from version 2.0.34)

• For each account, you need to specify a comma separated list of Splunk roles that are allowed to use this Remote account

• For retro-compatibility purposes, if the roles for an existing account have not been setup yet, TrackMe will use builtin roles in addition with typical admin roles (admin, sc_admin, trackme_user, trackme_power, trackme_admin)

• If the user calling the splunkremotesearch command is not a member of the account specified roles, access will be refused

• From version 2.0.61, both direct membership and inheritance are supported, users need to be a member any of the provided roles, or a role which inherites from any of these roles

Minimal RBAC requirements for the user account

TrackMe remote capabilities rely on a Splunk bearer token authentication, this token is associated with a Splunk user on the remote side which itself is associated with specific roles, capabilities, permissions and resources restrictions:

• Roles and capabilities: The user can be created with minimal permissions using the Splunk user role out of the box role. (you can inherit from user or a role providing the same capabilities than power)

• Indexes: Make sure the user can access to both normal and internal indexes.

• Restrictions: The user for TrackMe should not have any time limits restrictions, there are use cases which require long term searches.

• Resources: It is recommended to give to this user enough concurrent searches (unlike very basic or minimal user) as well as a sufficient quota. (5GB or 10Gb for instance)

Configuring a new remote account

When configuring a Splunk Remote deployment, you will provide:

• A name which uniquely identify the account (the account ID)

• The URL of the Splunk Search Head REST API, this can be Fully Qualified Domain Name, a hostname, an IP address (which itself can be a load balancer)

• Multiple REST endpoints can be submitted as a comma separated list of targets

• The bearer token value which is the secured credentials to access the environment, the bearer token is securely stored in the Splunk credential store

• The application name space where remote searches are to be executed, defaults to the Search application on the remote target

• A comma separated list of Splunk roles allowed to use this account (membership or inheritance)

• At the time of the creation or later on if you edit the account, a connectivity check is automatically performed which validates the network connectivity and authentication, if the check fails the interface will refuse the requested action

To configure a new remote account, access the Configuration user interface from the navigation bar:

If the connectivity check fails at the creation step, the configuration UI will raise an exception which indicates the root cause of the failure:

Once the account exists, you can test the connectivity easily using the following TrackMe REST endpoint in SPL:

| trackme mode=post url="/services/trackme/v2/configuration/test_remote_account" body="{'account': '<name of the account>'}"

The endpoint verifies that TrackMe can connect and authenticate successfully to the Splunk remote deployment.

When creating a new Virtual Tenant or a new Hybrid object, TrackMe will perform the same connectivity verification automatically:

Example of a Splunk Remote search performed by TrackMe

When performing Remote Splunk searches, TrackMe automatically defines an optimised search to split the search logic between the part executed remotely, and the search logic handled locally.

Example:

Using splunkremotesearch in a different application namespace

To use the command splunkremotesearch from a different application than TrackMe, such as Search & Reporting, you need to perform a few steps as follows.

Share TrackMe at system level

TrackMe by default is shared at the application level, you need to share at the system level as well as providing read access at least for your user roles: (“soc” in our example)

Manage capabilities

TrackMe requires capabilities to be able to access the command splunkremotesearch, the user roles need to have the capability:

• trackmeuseroperations

You can choose to inherit from the trackme_user role, or you can add the capability to your user roles as needed.

Note: Access is granted by any of the 3 builtin trackme_ user roles*

see: Role Based Access Control and ownership for more information about capabilities in TrackMe.

Manage Roles Based Access control on the account

User roles needs to be listed in the account, by default when creating an account, the following roles are listed:

• admin, sc_admin, trackme_user, trackme_power, trackme_admin

This is an explicit membership requirement, in our example we grant the user role “soc” to allow a member of this role to use the splunkremotesearch command against this account:

Use the command

In our example, the “soc” user is a member of a role with the same name, and can now use splunkremotesearch command as needed to perform searches against the remote deployment, from any other application namespace:

Accesses on the remote deployment rely on the service account which is associated with the bearer token configured in the account in TrackMe.

Troubleshooting failure to create or update a remote account

TrackMe Remote Search capabilities are an API based integration with Splunk API relying on Bearrer token authentication, you can refer to:

• https://docs.splunk.com/Documentation/Splunk/latest/Security/TroubleshootTokenAuth

When attempting to configure a new remote account, or update an existing account, TrackMe will attempt to validate the connectivity, both from a network and authentication perspective.

If this step fails, the UI will clearly raise and show the exception encountered, for instance:

Example of a network connectivity failure: (wrong host)

Example of a network connectivity failure: (timeout)

Example of a refused authentication by the remote counter part:

Note: You can also test the connectivity in pure SPL via the following TrackMe REST API endpoint:

| trackme mode=post url="/services/trackme/v2/configuration/test_remote_connectivity" body="{'target_endpoints': 'https://mysplunk.mydomain.com:8089', 'bearer_token': 'xxxx', 'app_namespace': 'search'}"

Inspect splunkd.log logs on the remote counter part:

If the connectivity fails for authentication related reasons, such as a wrong token or lack of capabilities, verify the splunkd.log on the remote counter part, you can use the following Splunk search:

index=_internal source=*/splunkd.log JsonWebToken

Messages will be logged by Splunk, eventually indicating the root cause of the failure.

Example of a refused token:

02-05-2024 17:30:44.226 +0000 ERROR JsonWebToken [2843158 TcpChannelThread] - JsonWebToken validation failed because: Token xxxx-xxxx has an invalid signature.