Configuration

Hint

Looking for a Quickstart?

• If you are looking for a quickstart, you can refer to the QUICK START - Starting with TrackMe: (feed tracking quickstart)

• The quickstart guide provides a step-by-step guide to get started with TrackMe, focusing on feed tracking, the easy and simple way

Using a service account for TrackMe is not mandatory

• It is not mandatory to use a service account for TrackMe, as described in the next steps, but it can be considered a good practice which provides several advantages in terms of management and monitoring

• However, this is not required as such and not doing so will not prevent TrackMe from operating normally

• By default, TrackMe will create knowledge objects and run searches as the “nobody” user (Splunk system account), which does not require any setup

Hint

Distinguishing permissions requirements between service accounts, TrackMe administrators, Power and read-only users

• A service account is a Splunk user (internal or SAML) that is used by TrackMe to perform scheduled activities, such as the creation of knowledge objects, the execution of scheduled searches, the creation of Virtual Tenants, etc.

• TrackMe users can have different levels of permissions. TrackMe comes with 3 built-in concepts for users: administrators, power users, and read-only users

• TrackMe leverages sophisticated techniques to ensure that you can define minimal permissions, for both the service account and the TrackMe users

• This allows for TrackMe users (TrackMe admins, power and read-only users) to avoid having to provide potentially dangerous capabilities such as list_settings, list_storage_passwords, etc.

• TrackMe comes with 3 built-in roles, trackme_admin, trackme_power and trackme_user

• trackme_user inherits from Splunk built-in user role, trackme_power inherits from trackme_user and trackme_admin inherits from trackme_power

• Each TrackMe built-in role enables the associated TrackMe capability, for instance trackme_admin enables the trackmeadminoperations capability

Summary Requirements for the TrackMe Service Account

The following requirements are the minimal requirements for the TrackMe service account:

Indexes Access

The service account should be able to search all non-internal indexes and all internal indexes (or at least the indexes containing data to be monitored, as well as the _internal index)

Capabilities

The service account should have the following capabilities at the minimum:

Hint

Inherit from Power role

• At the minimum, the service account should inherit from the Power role, which provides the basic required Splunk capabilities.

• Then add additional capabilities as needed.

Capability	Note
schedule_search	Required for running scheduled searches (provided by the Power role)
search	Required for running searches (provided by the Power role)
trackmeadminoperations	Can be inherited from the trackme_admin role (not provided by the Power role)
trackmepoweroperations	Can be inherited from the trackme_power role (not provided by the Power role)
trackmeuseroperations	Can be inherited from the trackme_user role (not provided by the Power role)

Summary Requirements for TrackMe Administrators

Essentially, TrackMe administrators need to have the following capability:

Capability	Note
trackmeadminoperations	Can be inherited from the trackme_admin role

In addition, and to be able to access and update the configuration menu items (Menu Configuration), administrators need:

• These are required capabilities by the Splunk UCC Framework, which is used by TrackMe for the purposes of the configuration level backend

• This is in fact optional; however, lacking these capabilities will not allow using the configuration UI to create remote service accounts for instance

• These capabilities would be required for users in charge of the highest level of administration of TrackMe; these are not required for the service accounts

• These capabilities are required only for the configuration UI; these are not required for the creation and/or management of TrackMe knowledge objects (virtual tenants, trackers, …)

Capability	Note
list_settings	Allows accessing the configuration UI
list_storage_passwords	Allows accessing the configuration UI
admin_all_objects	Allows updating the configuration items

Service Account and Permissions

To operate, TrackMe allows and recommends defining a Splunk user that has the ownership of any knowledge objects created by TrackMe as part of the Virtual Tenant lifecycle:

• Knowledge Objects (such as reports, alerts…) will be assigned to the user tagged as the owner of the Virtual Tenant

• Scheduled activities will run on behalf of the service account owner

By default, TrackMe assigns the user “admin” as the default owner of the Virtual Tenant; it is best practice to create your own service account owner. The following minimal permissions and capabilities are required:

• The service account needs to be a member of the built-in role trackme_admin as this provides the trackmeadminoperations capability, or this capability needs to be granted explicitly

• The service account needs to be able to search all non-internal indexes and all internal indexes

• The service account needs to be able to run scheduled searches; typically you can use the Splunk built-in power role

TrackMe implements a strict least privileges approach; consult Role Based Access Control and ownership

Note

Local service account user or SAML service account

• You can set up the service account user as a local user or a SAML user on the TrackMe Search Heads tier

• For other Search Head tiers, TrackMe can interact with the Splunk API for various powerful use cases such as TrackMe Flex Object trackers or the Workload component

• This requires a service account on the target Search Heads tier and a bearer token to be created

• If you want to create a SAML service account for TrackMe’s remote search capabilities, you need to have the SAML AQR setup, and an Identity Provider (IDP) supported by Splunk

• Reference: https://docs.splunk.com/Documentation/Splunk/latest/Security/Setupauthenticationwithtokens

• Reference: https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/SAMLConfigJWT

Creating a Service Account for TrackMe with Minimal Permissions

Warning

** permissions requirements for alert actions (stateful, notables…)**

• Make sure to give the following capabilities to the service account:

Capability	Note
list_settings	Allows accessing the configuration UI
list_storage_passwords	Allows accessing the configuration UI
admin_all_objects	Allows updating the configuration items

Note

Version 2.0.48 and later required for minimal permissions

• TrackMe version 2.0.48 and later is required for the following procedure allowing a strict minimalist service account

• Before this version, the service account needs to have extended capabilities such as list_settings and list_storage_passwords capabilities; therefore, the recommendation was for the service account to be a member of admin/sc_admin

• Some advanced use cases such as Flex Object trackers dealing with the Splunk | rest command or SOAR-related use cases may need additional capabilities to be granted to the service account user

One option is to create a specific role for the TrackMe service account with:

• Inheritance roles: power

• Role membership: trackme_admin

• Indexes: all non-internal and all internal indexes

• Resources: While TrackMe is optimized to distribute scheduled searches, it should be capable of running sufficient concurrent searches and it requires a large file quota to avoid issues

Hint

trackme_admin membership for the service account

• Before version 2.0.61, the service account needs to be an explicit member of the trackme_admin role (or the admin role in the tenants); this is needed because TrackMe requires explicit role membership (opposed to inheritance) to grant access to the Virtual Tenants

• From version 2.0.61, all RBAC dimensions in TrackMe support inheritance transparently

You can can then create the service account itself, example:

• The user is a member of the svc-trackme role

• As mentioned above, it is also a member of trackme_admin to be granted access to the Virtual Tenants

• Uncheck the box “Require password change on first login”

When you create a Virtual Tenant, you will specify the service account as the owner of the Virtual Tenant:

Hint

preset RBAC for the tenant creation UI

• Since the version 2.0.52, you can preset values for the owner and roles when creating a new Virtual Tenant from the UI

• Go in the Configuration then General Configuration

Minimal capabilities and resources for Remote Accounts and the user associated with the bearer token

TrackMe remote capabilities rely on a Splunk bearer token authentication, this token is associated with a Splunk user on the remote side which itself is associated with specific roles, capabilities, permissions and resources restrictions:

• Roles and capabilities: The user can be created with minimal permissions using the Splunk user role out of the box role. (you can inherit from user or a role providing the same capabilities than power)

• Indexes: Make sure the user can access to both normal and internal indexes.

• Restrictions: The user for TrackMe should not have any time limits restrictions, there are use cases which require long term searches.

• Resources: It is recommended to give to this user enough concurrent searches (unlike very basic or minimal user) as well as a sufficient quota. (5GB or 10Gb for instance)

• Additional capability required: Finally, for the purposes of the Workload, this user also needs to have the following capabilities granted admin_all_objects, select_workload_pool, list_workload_pools and list_workload_rules which are required for TrackMe’s backend to access to all objects of all applications in a remote manner. (for the Metadata in the Workload component)

In addition with the basic capabilities provided by the user role, these capabilities must be granted

Capability	Comment
admin_all_objects	Required
select_workload_pool	Required
list_workload_pools	Required
list_workload_rules	Required

Users and roles

TrackMe is deeply RBAC capable, consult the following documentation to configure users accesses for TrackMe:

• Role Based Access Control and ownership

Web Browsers and system compatibility

TrackMe should work fine with most Web Browsers and systems; however, if you experience icon issues due to the lack of support of ASCII emojis, you can enable the Bootstrap compatibility mode:

• Web Browser compatibility

Accessing TrackMe Configuration

TrackMe relies on the Splunk UCC Framework for the purposes of the configuration level backend:

• https://splunk.github.io/addonfactory-ucc-generator

The Splunk UCC framework provides various powerful features which are leveraged notably for the purposes of handling the application-level configuration; for these purposes, a configuration user interface is available:

Default configurations are located in the following configuration file:

trackme/default/trackme_settings.conf

The configuration can therefore be performed via:

• The configuration user interface: this creates a local/trackme_settings which is automatically replicated among the members when running in a Search Head Cluster

• By deploying a local/trackme_settings.conf accordingly (if running in Search Head Cluster, this file would be located in shcluster/apps/trackme/local/trackme_settings)

However, the recommended method as a basis is to configure TrackMe through the intended configuration user interface.

Remote Splunk deployments accounts

The Splunk remote deployments accounts tab is where you will configure any remote Splunk environment you will monitor with TrackMe:

Splunk remote deployment accounts are documented here: Splunk Remote Deployments (splunkremotesearch)

Virtual Tenants Accounts

Virtual Tenants Accounts are created and deleted automatically by TrackMe when managing Virtual Tenants through the Web or REST API; you can update the tenant-level configuration in this screen:

Email Delivery

THis tab defines emails delivery accounts, which can be used to connect to external SMTP servers to deliver emails notifications. (stateful alerts)

AI Provider

This tab defines Artificial Intelligence providers (LLM providers) that can be used for TrackMe’s AI related features:

General

This tab defines various general configuration:

Indexes general settings

This tab defines the indexes by default for Virtual Tenants:

If you intend to create Virtual Tenants specific indexes, we strongly recommend using a prefix pattern as a strict convention, for instance:

• trackme_<context>_<index name>

User interface configuration

This tab defines the user interface configuration:

splk-general

This tab defines various options specific to Splunk:

splk-data-sampling

This tab defines various options specific to the Data Sampling feature for splk-dsm (splk-feeds):

splk-outliers-detection

This tab defines various options specific to the Machine Outliers detection features:

SLA configuration

This tab defines various options specific to the SLA feature:

Maintenance

This tab defines various options specific to the Maintenance feature:

TrackMe Logging

This tab defines the logging level for TrackMe, all custom commands, REST endpoints, and any other TrackMe components rely on this setting to define the level of logging:

It is not recommended in a Production context to set TrackMe in DEBUG mode in normal circumstances as TrackMe will be extremely chatty in debug.

A typical logging message will look like: (INFO mode in this example)

2023-01-10 17:22:04,520 INFO trackmesplkflxparse.py stream 366 tenant_id="flx-demo-dma", context="live", TrackMeSplkFlxParse has terminated successfully, turn debug mode on for more details, results_count="2"

The logging level is extracted at search time, via props.conf settings, example:

# catch all sourcetype
[(?::){0}trackme:custom_commands:*]
EXTRACT-log_level = \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d*\s(?<log_level>\w*)\s

Therefore, you can review errors for instance with the following SPL search which would review both REST API endpoints errors and the custom commands:

(index=_internal sourcetype=trackme:rest_api log_level=ERROR) OR (index=_internal sourcetype=trackme:custom_commands:* log_level=ERROR)

We strongly believe that the truth stands in the logs; therefore, we take great care at making sure logging in TrackMe is giving you the greatest level of quality and reliability!

See the following documentation for more about logging & troubleshooting in TrackMe:

• Troubleshooting TrackMe