Configuration

Service Account and permissions

To operate, TrackMe allows and recommends to define a Splunk user that has the ownership of any knowledge objects created by TrackMe as part of the Virtual Tenant life cycle:

Knowledge Objects (such as reports, alerts…) will be assigned to the user tagged as the owner of the Virtual Tenant
Scheduled activities will run on behalf of the service account owner

By default, TrackMe assigns the user “admin” as the default owner of the Virtual Tenant, it is best practice to create your own service account owner, the following minimal permissions and capabilities are required:

The service account needs to be a member of the built in role trackme_admin as this provides the trackmeadminoperations capability, or this capability needs to be granted explicitly
The service account needs to be able to search all non internal indexes and all internal indexes
The service account needs to be able to run scheduled searches, typically you can use the Splunk built in power role

TrackMe implements a strict least privileges approach, consult Role Based Access Control and ownership

Note

local service account user or SAML service account

You can setup the service account user as a local user or a SAML user on the TrackMe Search Heads tier
For other Search Head tiers, TrackMe can interact with the Splunk API for various powerful use cases such as TrackMe Flex Object trackers or the Workload component
This requires a service account on the target Search Heads tier and a bearer token to be created
If you want to create a SAML service account for TrackMe’s remote search capabilities, you need to have the SAML AQR setup, and an Identity Provider (IDP) support by Splunk
Reference: https://docs.splunk.com/Documentation/Splunk/latest/Security/Setupauthenticationwithtokens
Reference: https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/SAMLConfigJWT

Creating a service account for TrackMe with minimal permissions

Note

Version 2.0.48 and later required for minimal permissions

TrackMe version 2.0.48 and later is required for the following procedure allowing a strict minimalist service account
Before this version, the service account needs to have extended capability such as list_settings and list_storage_passwords capabilities, therefore the recommendation was for the service account to be a member of admin/sc_admin
Some advanced use cases such as Flex Object trackers dealing with the Splunk | rest command or SOAR related use cases may need additional capabilities to be granted to the service account user

One option is to create a specific role for the TrackMe service account with:

Inheritance roles: power
Role membership: trackme_admin
Indexes: all non internal and all internal indexes
Resources: While TrackMe is optimised to distribute scheduled searches, it should be capable of running sufficient concurrent searches and it requires a large file quota to avoid issues

Hint

trackme_admin membership for the service account

Before the version 2.0.61, The service account needs to be an explicit member of the trackme_admin role (or the admin role in the tenants), this is needed because TrackMe requires explicit role membership (opposed to inheritance) to grant access to the Virtual Tenants
From the version 2.0.61, all RBAC dimensions in TrackMe support inheritance transparently

You can can then create the service account itself, example:

The user is a member of the svc-trackme role
As mentioned above, it is also a member of trackme_admin to be granted access to the Virtual Tenants
Uncheck the box “Require password change on first login”

When you create a Virtual Tenant, you will specify the service account as the owner of the Virtual Tenant:

Hint

preset RBAC for the tenant creation UI

Since the version 2.0.52, you can preset values for the owner and roles when creating a new Virtual Tenant from the UI
Go in the Configuration then General Configuration

General Configuration

TrackMe relies on the Splunk UCC Framework for the purposes of the configuration level backend:

https://splunk.github.io/addonfactory-ucc-generator

The Splunk UCC framework provides various powerful features which are leveraged notably for the purposes of handling the application level configuration, for this purposes a configuration user interface is available:

Default configuration are located in the following configuration file:

trackme/default/trackme_settings.conf

The configuration can therefore be performed via:

The configuration user interface: this creates a local/trackme_settings which is automatically replicated amongst the members when running in a Search Head Cluster
By deploying a local/trackme_settings.conf accordingly (if running in Search Head Cluster, this file would be located in shcluster/apps/trackme/local/trackme_settings)

However, the recommended method as a basis is to configure TrackMe through the intended configuration user interface.

Remote Splunk deployments accounts

The Splunk remote deployments accounts tab is where you will configure any remote Splunk environment you will monitor with TrackMe, if any,

Splunk remote deployment accounts are documented here: Splunk remote deployments (splunkremotesearch)

Configure accounts can be reviewed via the REST endpoint:

| rest splunk_server=local /servicesNS/nobody/trackme/trackme_account

TrackMe Logging

This tab defined the logging level for TrackMe, all custom commands, REST endpoints, and any other TrackMe components rely on this setting to define the level of logging:

It is not recommended in a Production context to set TrackMe in DEBUG mode in normal circumstances as TrackMe will be extremely chatty in debug.

A typical logging message will look like: (INFO mode in this example)

2023-01-10 17:22:04,520 INFO trackmesplkflxparse.py stream 366 tenant_id="flx-demo-dma", context="live", TrackMeSplkFlxParse has terminated successfully, turn debug mode on for more details, results_count="2"

The logging level is extracted at search time, via props.conf settings, example:

# catch all sourcetype
[(?::){0}trackme:custom_commands:*]
EXTRACT-log_level = \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d*\s(?<log_level>\w*)\s

Therefore, you can review errors for instance with the following SPL search which would review both REST API endpoints errors and the custom commands:

(index=_internal sourcetype=trackme:rest_api log_level=ERROR) OR (index=_internal sourcetype=trackme:custom_commands:* log_level=ERROR)

We strongly believe that the truth stands in the logs, therefore we take great care at making sure logging in TrackMe is giving you the greatest level of quality and reliability!

Indexes general settings

This tab defines the indexes by default for Virtual Tenants:

If you intend to create Virtual Tenants specific indexes, we strongly recommend to use a prefix pattern as a strict convention, for instance:

trackme_<context>_<index name>

Default Theme & Preferences Vtenants UI

You can define default theme preferences for the Virtual Tenant user interface, users can update these preferences for their own profile too:

General configuration

This tab defines various general configuration:

splk-general

This tab defines various options specific to Splunk:

splk-data-sampling

This tab defines various options specific to the Data Sampling feature for splk-dsm (splk-feeds):

splk-outliers-detection

This tab defines various options specific to the Machine Outliers detection features: