Configuration

Using a service account for TrackMe is not mandatory

• It is not mandatory to use a service account for TrackMe, as described in the next steps, but can be considered as a good practices which provides several advantages in term of management and monitoring

• However, this is not required as such and not doing so will not prevent TrackMe from operating normally

• By default, TrackMe will create knowledge objects and run searches as the “nobody” user (Splunk system account), which does not require any setup

Hint

Distinguishing permissions requirements between service accounts, TrackMe administators, Power and read only users

• A service account is a Splunk user (internal or SAML) that is used by TrackMe to perform scheduled activities, such as the creation of knowledge objects, the execution of scheduled searches, the creation of Virtual Tenants, etc…

• TrackMe users can have difference levels of permissions, TrackMe comes with 3 buitin concepts for users, administrators, power users and read only users

• TrackMe leverages sophisticated techniques to ensure that you can define minimal permissions, for both the service account and the TrackMe users

• This allows for TrackMe users (TrackMe adnins, power and read only users) to avoid having to provide potentially dangerous capabilities such as list_settings, list_storage_passwords, etc…

• TrackMe comes with 3 builtin roles, trackme_admin, trackme_power and trackme_user

• trackme_users inherites from Splunk builtin user role, trackme_power inherits from trackme_user and trackme_admin inherits from trackme_power

• Each TrackMe builtin role enables the associated TrackMe capability, for instance trackme_admin enables the trackmeadminoperations capability

Summary requirements for the TrackMe service account

The following requirements are the minimal requirements for the TrackMe service account:

Indexes accesses

The service account should be able to search all non internal indexes and all internal indexes (or at least the indexes containing data to be monitored, as well as the _internal index)

Capabilities

The service account should have the following capabilities at the minimal:

capability	note
get_metadata
get_typeahead
list_accelerate_search
list_all_objects
list_inputs
list_metrics_catalog
output_file
pattern_detect
request_remote_tok
rest_access_server_endpoints
rest_apps_view
rest_properties_get
rest_properties_set
run_collect
run_custom_command
run_dump
run_mcollect
schedule_search	required for running scheduled searches
search
trackmeadminoperations	can be inherited from the trackme_admin role
trackmepoweroperations	can be inherited from the trackme_power role
trackmeuseroperations	can be inherited from the trackme_user role

Summary requirements for TrackMe administators

Essentially, TrackMe administrators need to have the following capability:

capability	note
trackmeadminoperations	Can be inherited from the trackme_admin role

In Addition, and to be able to access and update the configuration menu items (Menu Configuration), administrators need:

• These are required capabilities by the Splunk UCC Framework, which is used by TrackMe for the purposes of the configuration level backend

• This is in fact optional, however lacking these capabilities will not allow using the configuration UI to create remote service accounts for instance

• These capabilities would be required for users in charge of the highest level of administration of TrackMe, these are not required for the service accounts

• These capabilities are required only for the configuration UI, these are not required for the creation and/or management of TrackMe knowledge objects (virtual tenants, trackers, …)

capability	note
list_settings	Allows accessing the configuration UI
list_storage_passwords	Allows accessing the configuration UI
admin_all_objects	Allows updating the configuration items

Service Account and permissions

To operate, TrackMe allows and recommends to define a Splunk user that has the ownership of any knowledge objects created by TrackMe as part of the Virtual Tenant life cycle:

• Knowledge Objects (such as reports, alerts…) will be assigned to the user tagged as the owner of the Virtual Tenant

• Scheduled activities will run on behalf of the service account owner

By default, TrackMe assigns the user “admin” as the default owner of the Virtual Tenant, it is best practice to create your own service account owner, the following minimal permissions and capabilities are required:

• The service account needs to be a member of the built in role trackme_admin as this provides the trackmeadminoperations capability, or this capability needs to be granted explicitly

• The service account needs to be able to search all non internal indexes and all internal indexes

• The service account needs to be able to run scheduled searches, typically you can use the Splunk built in power role

TrackMe implements a strict least privileges approach, consult Role Based Access Control and ownership

Note

local service account user or SAML service account

• You can setup the service account user as a local user or a SAML user on the TrackMe Search Heads tier

• For other Search Head tiers, TrackMe can interact with the Splunk API for various powerful use cases such as TrackMe Flex Object trackers or the Workload component

• This requires a service account on the target Search Heads tier and a bearer token to be created

• If you want to create a SAML service account for TrackMe’s remote search capabilities, you need to have the SAML AQR setup, and an Identity Provider (IDP) support by Splunk

• Reference: https://docs.splunk.com/Documentation/Splunk/latest/Security/Setupauthenticationwithtokens

• Reference: https://docs.splunk.com/Documentation/SplunkCloud/latest/Security/SAMLConfigJWT

Creating a service account for TrackMe with minimal permissions

Note

Version 2.0.48 and later required for minimal permissions

• TrackMe version 2.0.48 and later is required for the following procedure allowing a strict minimalist service account

• Before this version, the service account needs to have extended capability such as list_settings and list_storage_passwords capabilities, therefore the recommendation was for the service account to be a member of admin/sc_admin

• Some advanced use cases such as Flex Object trackers dealing with the Splunk | rest command or SOAR related use cases may need additional capabilities to be granted to the service account user

One option is to create a specific role for the TrackMe service account with:

• Inheritance roles: power

• Role membership: trackme_admin

• Indexes: all non internal and all internal indexes

• Resources: While TrackMe is optimised to distribute scheduled searches, it should be capable of running sufficient concurrent searches and it requires a large file quota to avoid issues

Hint

trackme_admin membership for the service account

• Before the version 2.0.61, The service account needs to be an explicit member of the trackme_admin role (or the admin role in the tenants), this is needed because TrackMe requires explicit role membership (opposed to inheritance) to grant access to the Virtual Tenants

• From the version 2.0.61, all RBAC dimensions in TrackMe support inheritance transparently

You can can then create the service account itself, example:

• The user is a member of the svc-trackme role

• As mentioned above, it is also a member of trackme_admin to be granted access to the Virtual Tenants

• Uncheck the box “Require password change on first login”

When you create a Virtual Tenant, you will specify the service account as the owner of the Virtual Tenant:

Hint

preset RBAC for the tenant creation UI

• Since the version 2.0.52, you can preset values for the owner and roles when creating a new Virtual Tenant from the UI

• Go in the Configuration then General Configuration

Minimal capabilities and resources for Remote Accounts and the user associated with the bearer token

TrackMe remote capabilities rely on a Splunk bearer token authentication, this token is associated with a Splunk user on the remote side which itself is associated with specific roles, capabilities, permissions and resources restrictions:

• Roles and capabilities: The user can be created with minimal permissions using the Splunk user role out of the box role. (you can inherit from user or a role providing the same capabilities than power)

• Indexes: Make sure the user can access to both normal and internal indexes.

• Restrictions: The user for TrackMe should not have any time limits restrictions, there are use cases which require long term searches.

• Resources: It is recommended to give to this user enough concurrent searches (unlike very basic or minimal user) as well as a sufficient quota. (5GB or 10Gb for instance)

• Additional capability required: Finally, for the purposes of the Workload, this user also needs to have the following capabilities granted admin_all_objects, select_workload_pool, list_workload_pools and list_workload_rules which are required for TrackMe’s backend to access to all objects of all applications in a remote manner. (for the Metadata in the Workload component)

In addition with the basic capabilities provided by the user role, these capabilities must be granted

Capability	Comment
admin_all_objects	Required
select_workload_pool	Required
list_workload_pools	Required
list_workload_rules	Required

Users and roles

TrackMe is deeply RBAC capable, consult the following documentation to configure users accesses for TrackMe:

• Role Based Access Control and ownership

Web Browsers and system compatibility

TrackMe should just waork fine with most Web Browsers and systems, however if you experience icons issues due to the lack of support of ascii emojis, you can enable the Bootstrap compatibility mode:

• Web Browser compatibility

Accessing TrackMe Configuration

TrackMe relies on the Splunk UCC Framework for the purposes of the configuration level backend:

• https://splunk.github.io/addonfactory-ucc-generator

The Splunk UCC framework provides various powerful features which are leveraged notably for the purposes of handling the application level configuration, for this purposes a configuration user interface is available:

Default configuration are located in the following configuration file:

trackme/default/trackme_settings.conf

The configuration can therefore be performed via:

• The configuration user interface: this creates a local/trackme_settings which is automatically replicated amongst the members when running in a Search Head Cluster

• By deploying a local/trackme_settings.conf accordingly (if running in Search Head Cluster, this file would be located in shcluster/apps/trackme/local/trackme_settings)

However, the recommended method as a basis is to configure TrackMe through the intended configuration user interface.

Remote Splunk deployments accounts

The Splunk remote deployments accounts tab is where you will configure any remote Splunk environment you will monitor with TrackMe, if any,

Splunk remote deployment accounts are documented here: Splunk remote deployments (splunkremotesearch)

Remote Splunk Deployments Accounts

Field	Default Value	Description
name		Enter a unique name for this Splunk remote environment. (lower case alphanumeric characters and underscores)
splunk_url		A list of comma separated list of targets, ex: https://splunk1:8089,https://splunk2:8089 (SSL is enforced and URLs will be prefixed with https:// if not set), the URL can be based on IP or FQDN
bearer_token		For client instances, set the bearer token used for remote access to the KVstore instance
app_namespace	search	The Splunk application namespace on the remote system where searches will be executed, defaults to the Splunk search app
rbac_roles	admin,sc_admin,trackme_user,trackme_power,trackme_admin	A comma separated list of Splunk roles that are allowed to access this account, either by direct membership or by inheritance

Virtual Tenants Accounts

Virtual Tenants Accounts are created and deleted automatically by TrackMe when maanaging Virtual Tenants throught the Web or REST API, you can uopdate the tenant level configuration in this screen:

Virtual Tenants Preferences

Field	Default Value	Description
name		The main identifier for the tenant (Tenant ID)
description		Enter the Virtual Tenant description
ui_min_object_width	300	UI tenant preferences - for all component - the minimal width in pixels for the object field, increase this value to handle long entity names
ui_expand_metrics	0	UI tenant preferences - for all eligible components - expand metrics information by default or rely on the right click context menu
outliers_set_state	1	Feature tenant behaviour - for all eligible components - allows Outliers to influence entities status for this tenant
data_sampling_set_state	1	Feature tenant behaviour - for splk-dsm only - allows data sampling to influence entities status for this tenant
data_sampling_obfuscation	0	Feature tenant behaviour - for splk-dsm only - when generating Data samples, you can use this option to avoid storing clear text copies of sampled events in the KVstore
cmdb_lookup	1	CMDB lookup integration - Enable or disable the UI CMDB integration for this tenant

General configuration

This tab defines various general configuration:

TrackMe General Configuration

Field	Label	Default Value	Description
trackme_default_sharing	Default sharing level	app	When TrackMe creates knowledge objects, define if the sharing level should be app or global
trackme_owner_default	Default owner	admin	The default owner user preset in the dropdown of the Virtual Tenants creation user interfaces when creating new Virtual Tenants
trackme_admin_role_default	Default admin role	trackme_admin	The default admin role preset in the dropdown of the Virtual Tenants creation user interfaces when creating new Virtual Tenants
trackme_power_role_default	Default power role	trackme_power	The default power role preset in the dropdown of the Virtual Tenants creation user interfaces when creating new Virtual Tenants
trackme_user_role_default	Default user role	trackme_user	The default user role preset in the dropdown of the Virtual Tenants creation user interfaces when creating new Virtual Tenants
state_events_minimal	Minimal state events	1	Influences the volume of information in state events; in minimal mode, only key information is retained to limit size and costs
state_events_allowlist	allowlist fields (minimal)	alias,anomaly_reason,keyid…	Comma-separated list of fields allowed in state events when in minimal mode
state_events_blocklist	In full, block list fields	_raw,info_max_time,info_min_time…	Comma-separated list of fields blocked in state events when in full mode
allow_admin_operations	Allow admin operations	1	Allow TrackMe UI to show accesses to admin level operations, such as the creation and management of Virtual Tenants
enable_conf_manager_receiver	Enable TrackMe Conf Manager	0	Enables the TrackMe conf manager receiver; admin level operations will be sent to the receiver to be replayed in the target environment
trackme_ack_duration_default	Default Ack duration	86400	Default duration in seconds proposed when opening the acknowledgment action for an entity in alert
trackme_ack_remove_when_green	Remove Ack behaviour	1	Automatically removes an Acknowledgment when the entity is back in green state, even if it has not expired yet (unless sticky Ack)
trackme_future_tolerance	Future indexing tolerance	-600	Amount in negative seconds used for tolerance before assuming data is indexed in the future
trackme_auto_disablement_period	Auto disablement period	45d	Period in relative days, after which an inactive entity (not sending data) gets disabled automatically

Indexes general settings

This tab defines the indexes by default for Virtual Tenants:

If you intend to create Virtual Tenants specific indexes, we strongly recommend to use a prefix pattern as a strict convention, for instance:

• trackme_<context>_<index name>

Indexes General Settings

Field	Default Value	Description
trackme_idx_search_filter	trackme_*	An index pattern which matches all TrackMe index, for instance trackme_* which matches all indexes starting by the prefix trackme
trackme_notable_idx	trackme_notable	This index will be used to store TrackMe notable events, this can be overridden per tenant
trackme_summary_idx	trackme_summary	This index will be used to store all summary events generated, this can be overridden per tenant
trackme_metric_idx	trackme_metrics	This index will be used to store metrics, this can be overridden per tenant
trackme_audit_idx	trackme_audit	This index will be used to store audit events, this can be overridden per tenant

Prefs Vtenants UI

You can define default theme preferences for the Virtual Tenant user interface, users can update these preferences for their own profile too:

Prefs Home UI

You can define default theme preferences for the Home user interface:

splk-general

This tab defines various options specific to Splunk:

splk-general Configuration

Field	Default Value	Description
splk_general_idx_filter	host=*	Search filter for views inspecting the indexed time activity such as line breaking issues or datetime parsing, filter on indexers and/or heavy forwarders, example: host=idx*.splunkcloud.com
splk_general_dsm_threshold_default	3600	The default latency threshold value in seconds applied for splk-dsm based entities, defines the maximum allowed value in seconds for ingestion latency
splk_general_dsm_delay_default	3600	The default delay threshold value in seconds applied for splk-dsm based entities, defines the maximum allowed value in seconds for the delay of the entity
splk_general_dhm_threshold_default	3600	The default latency threshold value in seconds applied for splk-dhm based entities, defines the maximum allowed value in seconds for ingestion latency
splk_general_dhm_delay_default	3600	The default delay threshold value in seconds applied for splk-dhm based entities, defines the maximum allowed value in seconds for the delay of the entity
splk_general_mhm_threshold_default	900	The default threshold value in seconds applied for splk-mhm based entities, defines the maximal threshold defined by default for entities in the splk-mhm component (metrics delay)
splk_general_elastic_max_concurrent	3	System level number of parallel concurrent searches for Shared Elastic sources, this can be overriden on a per tenant basis using max_concurrent_searches on the Shared Elastic tracker
splk_general_dsm_cmdb_search	inputlookup my_cmdb where (index="$data_index$" AND sourcetype="$data_sourcetype$")	The CMDB lookup search for this component, you can refer using tokens to any field that TrackMe maintains in the KVstore collection
splk_general_dhm_cmdb_search	inputlookup my_cmdb where (host="$alias$")	The CMDB lookup search for this component, you can refer using tokens to any field that TrackMe maintains in the KVstore collection
splk_general_mhm_cmdb_search	inputlookup my_cmdb where (host="$alias$")	The CMDB lookup search for this component, you can refer using tokens to any field that TrackMe maintains in the KVstore collection, use the token $tenant_id$ to make this search tenant specific
splk_general_cim_cmdb_search	inputlookup my_cmdb where (object="$object$")	The CMDB lookup search for this component, you can refer using tokens to any field that TrackMe maintains in the KVstore collection, use the token $tenant_id$ to make this search tenant specific
splk_general_flx_cmdb_search	inputlookup my_cmdb where (object="$object$")	The CMDB lookup search for this component, you can refer using tokens to any field that TrackMe maintains in the KVstore collection, use the token $tenant_id$ to make this search tenant specific
splk_general_wlk_cmdb_search	inputlookup my_cmdb where (savedsearch_name="$savedsearch_name$")	The CMDB lookup search for this component, you can refer using tokens to any field that TrackMe maintains in the KVstore collection, use the token $tenant_id$ to make this search tenant specific
splk_general_dsm_docs_note_global	No default value	In TrackMe’s UI, you can reference a documentation note and link per entity, use this option to define a global default documentation note which can still be overriden on per entity basis but will appear as a default
splk_general_dsm_docs_link_global	No default value	In TrackMe’s UI, you can reference a documentation note and link per entity, use this option to define a global default documentation link which can still be overriden on per entity basis but will appear as a default

splk-data-sampling

This tab defines various options specific to the Data Sampling feature for splk-dsm (splk-feeds):

splk-data-sampling Configuration

Field	Default Value	Description
splk_data_sampling_default_sample_record_at_discovery	100	Number of events to sample the first time the data sample engine runs against a given entity. Increased value improves the event format recognition during the discovery
splk_data_sampling_default_sample_record_at_run	50	Number of events to sample by default during the engine execution. Increased value improves the event format recognition at the cost of more records stored in the KVstore collection

splk-outliers-detection

This tab defines various options specific to the Machine Outliers detection features:

splk-outliers-detection Configuration

Field	Default Value	Description
splk_outliers_time_train_mlmodels_default	86400	The time value in seconds requested for ML models to be trained for entities, a given entity will regularly get ML models trained if possible based on this value. (defaults to 1 day)
splk_outliers_time_monitor_mlmodels_default	3600	The time value in seconds requested for ML models to be monitored for entities, a given entity will regularly get ML models monitored if possible based on this value. (defaults to 1 hour)
splk_outliers_max_runtime_train_mlmodels_default	900	The time value in seconds requested to limit the max duration of the ML training models, defaults to 15 min (reduced by 30 sec) and should be set according to the cron schedule of the ML training job
splk_outliers_detection_disable_default	0	When a new entity is discovered, enable or disable the volume based outliers detection by default for that entity. The feature can still be managed on demand for that entity.
splk_outliers_calculation_default	stdev	The default calculation mode used for anomaly outliers detection, can be updated per entity.
splk_outliers_density_lower_threshold_default	0.005	The default value of the lower threshold applied to the DensityFunction algorithm, set at discovery and be updated per entity.
splk_outliers_density_upper_threshold_default	0.005	The default value of the upper threshold applied to the DensityFunction algorithm, set at discovery and be updated per entity.
splk_outliers_alert_lower_threshold_volume_default	1	Alert when the lower bound threshold is breached for volume based KPIs.
splk_outliers_alert_upper_threshold_volume_default	0	Alert when the upper bound threshold is breached for volume based KPIs.
splk_outliers_alert_lower_threshold_latency_default	0	Alert when the lower bound threshold is breached for volume based KPIs.
splk_outliers_alert_upper_threshold_latency_default	1	Alert when the upper bound threshold is breached for latency based KPIs.
splk_outliers_detection_period_default	-30d	The relative period used by default for outliers calculations, applied during entity discovery and can be updated per entity.
splk_outliers_detection_timefactor_default	-30d	The default time factor applied for the outliers dynamic thresholds calculation.
splk_outliers_detection_latency_kpi_metric_default	splk.feeds.avg_latency_5m	The default kpi metric for latency outliers detection.
splk_outliers_detection_volume_kpi_metric_default	splk.feeds.avg_eventcount_5m	The default kpi metric for volume outliers detection.
splk_outliers_auto_correct	1	When defining the model, enable or disable auto_correct by default, which uses the concept of auto correction based on min lower and upper deviation.
splk_outliers_perc_min_lowerbound_deviation_default	5.0	If an outlier is not deviant (LowerBound) from at least that percentage of the current KPI value, it will be considered as a false positive.
splk_outliers_perc_min_upperbound_deviation_default	5.0	If an outlier is not deviant (UpperBound) from at least that percentage of the current KPI value, it will be considered as a false positive.

TrackMe Logging

This tab defines the logging level for TrackMe, all custom commands, REST endpoints, and any other TrackMe components rely on this setting to define the level of logging:

It is not recommended in a Production context to set TrackMe in DEBUG mode in normal circumstances as TrackMe will be extremely chatty in debug.

A typical logging message will look like: (INFO mode in this example)

2023-01-10 17:22:04,520 INFO trackmesplkflxparse.py stream 366 tenant_id="flx-demo-dma", context="live", TrackMeSplkFlxParse has terminated successfully, turn debug mode on for more details, results_count="2"

The logging level is extracted at search time, via props.conf settings, example:

# catch all sourcetype
[(?::){0}trackme:custom_commands:*]
EXTRACT-log_level = \d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\,\d*\s(?<log_level>\w*)\s

Therefore, you can review errors for instance with the following SPL search which would review both REST API endpoints errors and the custom commands:

(index=_internal sourcetype=trackme:rest_api log_level=ERROR) OR (index=_internal sourcetype=trackme:custom_commands:* log_level=ERROR)

We strongly believe that the truth stands in the logs, therefore we take great care at making sure logging in TrackMe is giving you the greatest level of quality and reliability!

See the following documentation for more about looging & troubleshooting in TrackMe:

• Troubleshooting