Workload (splk-wlk) - Manage Workload tenants and trackers

Introduction to the Workload component

The TrackMe workload component (splk-wlk) is a licensed users restricted component which tracks the activity from any kind of scheduled search in your Splunk deployment, from reports to alerts and Data Model Acceleration. (DMA)

The Workload component tracks various use cases such as execution errors which can slightly affect the consistency of your Splunk use cases, providing powerful features to detect issues and track the performance of the Splunk scheduling activity.

For the User Guide, consult: Splunk Workload (splk-wlk)

Creating and configuring a Workload Tenant

The creation and configuration of a Workload Tenant is guided through the wizard, click on the “create new tenant” and choose the associated component to start the configuration process:

screen1.png

Workload component in Splunk Cloud

Generally, in Splunk Cloud have two or more Search Head layers:

  • The ad-hoc Search Head tier, where TrackMe should be used (standalone or SHC)

  • The premium app Search Head tier, running Enterprise Security or ITSI (standalone or SHC)

In a nutshell, the configuration path is the following:

  • configure a TrackMe Splunk Remote Deployment which targets the premium app Search Head tier (this is required notably for the versioning tracking of scheduled searches, even through both Search Head tiers access the same indexing tier)

  • create a new Splunk eventtype and tag in the premium app Search Head tier which will be used in TrackMe to restrict the Workload tracking activity to this Search Head tier

  • create and configure the Workload tenant in TrackMe

Step 1: configure the remote Splunk Deployment

For more information about TrackMe’s Splunk Remote Deployments capabilities and configuration, consult: Splunk remote deployments (splunkremotesearch)

Proceed as following:

  • Create a bearer token in the premium app Search Head tier, it is good practice to associate the bearer token with a service account dedicated for that purpose

screen2.png

  • Identify the Search Head tiers members:

index=_internal sourcetype=scheduler host=sh*.splunkcloud.com
| stats values(app) as app by host
| eval is_es=if(match(app, "(ESS|SA-AccessProtection|SA-EndpointProtection|SA-NetworkProtection|SA-ThreatIntelligence|EnterpriseSecuritySuite)"), "true", "false")
| stats values(host) as host by is_es

note: Search Heads can technically change although this is very rare (such as for migration purposes), if Search tier members change, the tag should be updated

For example, the eventtype and tag will be called shc_es, the eventtype will match the Splunk Cloud ES search head members against the _internal:

host="shxxx1.splunkcloud.com" OR  host="shxxx2.splunkcloud.com" OR host="shxxx3.splunkcloud.com"

Warning

The tag needs to be created **on the target Search Head tier, not on the ad-hoc Search Head tier running TrackMe.

  • Create the eventtype and tag in the ES search head

  • You can create in any namespace as long as the eventtype and tag can be read through the splunkremotesearch account that was created

eventtypes.png

Once you have validated the connectivity, you can continue and start the configuration of the Workload tenant.

Step 2: create and configure the Workload tenant

Open the assistant wizard, and proceed as follows:

  • Select “Splunk Cloud” in the Splunk environment dropdown:

screen3.png

  • Select the Remote deployment target:

screen4.png

  • Set the search restrictions:

Hint

Search restrictions per Workload component

  • Since the release 2.0.57, you can define the search restrictions per component, to ease and improve the configuration process

  • You can directly validate that the search will return results by clicking on the search preview link

  • This opens a pre-populated search interface, it also sets the restrictions and calls splunkremotesearch if the search has to run against a remote Search Head tier

screen5.png

Complete the rest of the setup, it can take up to 5 minutes before entities start to be visible in the Workload tenant.

Workload component in Splunk Enterprise

If you are running Splunk Enterprise, then the setup slightly depends on your deployment:

  • If you have multiple Search Head tiers, you can use the same tag approach as documented in the Splunk Cloud setup above

  • You can also create a custom indexed or search time field on the Search Head tier level, which automatically identifies the Search Head tiers to be targeted

  • Use different combinations up to your preference

Managing Workload Trackers

There are a certain number of Workload trackers which are created automatically by the wizard, depending on your Splunk environment.

Through the trackers management user interface, you can eventually add new trackers if needed, either if you want to re-create the trackers or extend the tracking to additional context:

screen6.png

However, the general recommendation is to dedicate a Workload Tenant to a given Search Head tier.

The REST API UI also allows you to manage all types of Workload trackers:

screen7.png

Managing Workload detection

When a Workload Virtual Tenant is created, TrackMe automatically creates a macro which defines the main status definition behaviour for that tenant.

The macro is named as follows:

  • trackme_wlk_set_status_tenant_<tenant_id>

You can update this macro to modify how TrackMe defines the status of Workload entities, for instance these lines define the grace time period for the detection of delayed entities:

``` Calculate the delta in sequence between now and the last execution compared against the requested cron schedule sequence, add 1h of grace time, detect if the execution has been delayed ```
| eval status=if(cron_exec_sequence_sec>0 AND ( now()-last_seen > (cron_exec_sequence_sec + 3600) ), 2, status)

This also defines the level of skipping searches (percentage) which leads entities to turn orange or red:

``` If there are skipping searches, define two levels of alerting, less than 5% is 3 (orange), more is 2 (red) ```
| eval status=case(skipped_pct>0 AND skipped_pct<5, 3, skipped_pct>0 AND skipped_pct>=5, 2, 1=1, status)

The macro is part of the life cycle of the tenant’s knowledge objects, therefore it will be removed automatically if the tenant itself is deleted.

This macro is executed by the “main” tracker, which naming convention is the following:

  • trackme_wlk_hybrid_main_ce555e75db1643a_wrapper_tenant_<tenant_id>