Performing mass operations in TrackMe

Introduction to mass operations in TrackMe

About mass operations in TrackMe

  • There are different contexts in which mass operations can be performed in TrackMe.

  • Depending on the context, you can perform mass operations directly in the TrackMe UI (bulk edit), in some contexts you can also directly use TrackMe REST API endpoints and eventually take advantages of Splunk SPL techniques.

  • The purpose of this whitepaper is to document various typical mass operations that can be performed in TrackMe, and to provide guidance on how to perform them.

Mass deleting entities

The following example demoes how to mass delete entities in TrackMe:

Hint

Replace tenant and component name:

  • Replace “mytenant” with the tenant ID you want to delete entities from.

  • “dsm” is the component name, simply replace it with the component you want to delete entities from. (dsm/dhm/mhm/wlk/flx)

Context:

  • In this example, we assume that we have a large list of entities that should be removed from splk-dsm, for instance due to an issue in Splunk which has lead to the unwanted creation of a large number of sourcetypes.

Step 1: Get entities

| inputlookup trackme_dsm_tenant_mytenant | eval keyid=_key
``` in this example, we want to delete all entities related to the main index, which unexpecty was fed with a large number of unwanted entities ```
| search data_index="main"

Step 2: Collect keys and turn into a CSV list

| inputlookup trackme_dsm_tenant_mytenant | eval keyid=_key
``` in this example, we want to delete all entities related to the main index, which unexpecty was fed with a large number of unwanted entities ```
| search data_index="main"
| table keyid
| stats values(keyid) as keys
| eval keys=mvjoin(keys, ",")

Step final: Delete entities

| inputlookup trackme_dsm_tenant_mytenant | eval keyid=_key
``` in this example, we want to delete all entities related to the main index, which unexpecty was fed with a large number of unwanted entities ```
| search data_index="main"
| table keyid
| stats values(keyid) as keys
| eval keys=mvjoin(keys, ",")
``` finally call the endpoint, which supports multiple entities deletion using the Splunk map command```
| map search="| trackme url=/services/trackme/v2/splk_dsm/write/ds_delete mode=post body=\"{'tenant_id':'mytenant', 'deletion_type': 'temporary', 'keys_list':'$keys$'}\""

Run the search above, if dealing with a very large number of entities, this search can take some time to complete, send the search to background and monitor the progress.

Mass enabling/disabling entities

Here is a customer use case:

  • TrackMe is configured to automatically disable the monitoring state when new entities are discovered.

  • The purpose is to avoid creating noise until we have enough historical knowledge of the feeds behaviours, so that TrackMe’s features such as the adaptive delay will eventually adapt the delay values if needed.

  • Then, a custom schedule search inspects entities, when we have reached a state of having enough historical knowledge, disabled entities are enabled automatically.

Configuring TrackMe to disable the monitoring status at the time of the feed discovery:

The default monitored state is handled via a Splunk macro in TrackMe, which definition in macros.conf is as follows:

[trackme_default_monitored_state]
definition = eval monitored_state=if(isnull(monitored_state), "enabled", monitored_state)
iseval = 0

To disable the monitoring state at the time of the feed discovery, you can override the macro definition to:

eval monitored_state=if(isnull(monitored_state), "disabled", monitored_state)

Creatting a scheduled report which leverages TrackMe’s metrics to define a confidence status, then call map and TrackMe’s REST API to enable entities:

The following search would be scheduled at your convenience, and handle entities enabling based on a confidence status:

Hint

Replace tenant:

  • Replace “mytenant” with the tenant ID value.

| mstats latest(trackme.splk.feeds.lag_event_sec) as lag_event_sec where index=trackme_metrics tenant_id="mytenant" object_category="splk-dsm" by object span=1d

| stats min(_time) as first_time by object
| eval metrics_duration=now()-first_time
| eval confidence=if(metrics_duration<(7*86400), "low", "normal")
| where confidence="normal"

| lookup trackme_dsm_tenant_mytenant object OUTPUT _key as keyid, monitored_state, priority
| where (isnotnull(keyid) AND monitored_state="disabled")

|  stats values(keyid) as keys
|  eval keys=mvjoin(keys, ",")

|  map search="| trackme url=\"/services/trackme/v2/splk_dsm/write/ds_monitoring\" mode=\"post\" body=\"{'tenant_id': 'mytenant', 'action': 'enable', 'keys_list': '$keys$'}\""

Maintaining entities priority in a Splunk lookup file

A frequent question from TrackMe users is how can they maintain the priority of entities in a lookup file, and how to update the priority of entities in bulk.

In this example, we will demonstrate how to update the priority of entities in bulk for splk-dhm (Data Hosts tracking) component.

Step 1: Prepare the lookup file

Create a lookup file which contains at the minimum the following fields:

  • host: the machine hostname as it is seen in Splunk. (the host metadata)

  • priority: the priority value you want to set for the entity.

While it is not fully mandatory, calling the lookup through a lookup definition is recommended so you can run a case insensitive lookup call, you should be able to run:

| inputlookup hosts_lookup_cmdb

About case sensitive match and lookup share

  • Ensure to create a definition for the lookup so you can configure Splunk to allow case insensitive lookup matches.

  • Also, make sure both the lookup file and lookup permissions are set to be shared so TrackMe can access these.

Step 2: Create a report that will be scheduled and will update and maintain the priority of entities based on the lookup content

This is really simple, TrackMe stores the priority in the main KVstore collection, updating it based on a third party lookup can therefore be handled through a simple SPL logic:

Hint

Replace tenant:

  • Replace “mytenant” with the tenant ID value.

Hint

About the alias field and components:

  • In the example below, we use the TrackMe internal field called alias.

  • In the context of splk-dhm, the alias will be the exact host value as seen in Splunk.

  • In fact, TrackMe stores the unique object value in a field called object and is prefixed with a TrackMe metadata key:`, which is why we use the alias field instead to perform the lookup call.

| inputlookup trackme_dhm_tenant_mytenant | eval keyid=_key
| lookup hosts_lookup_cmdb host as alias OUTPUT priority as lookup_priority
| table keyid, lookup_priority, priority, *
| where isnotnull(lookup_priority)
| where priority!=lookup_priority
``` At this stage only entities to be managed will be updated ```
| eval priority=lookup_priority, mtime=now()
``` Update ```
| outputlookup append=t key_field=keyid trackme_dhm_tenant_mytenant

Save this search as a report and schedule it at your convenience, it doesn’t need to be executed very often so once every 4 hours for instance is likely enough.

Finally, note that this report will not return any results unless a priority update is needed, not returning any results simply indicates that all entities are up to date according to the lookup file.

Maintaining priority through a third party lookup would override any change made through the UI or REST API:

  • Note that this logic above would override a priority change a TrackMe power user would make through the UI or REST API.

  • Therefore, this needs to be known and documented to avoid any risks of confusion.

About other components:

  • The logic or variations of this logic can be applied to any TrackMe component.

  • Simply replace the component suffix, in this example _dhm with the associated component suffix. (dsm, mhm, flx, wlk, cim)

Mass updating entities priority

The following example demoes how to mass update entities priority in TrackMe:

Hint

Replace tenant:

  • Replace “mytenant” with the tenant ID value.

  • Replace the component, in this example we use splk-dsm, replace it with the component you want to update the priority for. (notably the KVstore collection prefix, dsm/dhm/mhm/flx/wlk/cim)

Step 1: Get entities

This example updates entities associated with indexes starting with “org”, replace for your context

| inputlookup trackme_dsm_tenant_mytenant | eval keyid=_key
``` in this example, we want to update all entities related to indexes starting by org* ```
| search data_index="org*"

Step 2: Collect keys and turn into a CSV list

| inputlookup trackme_dsm_tenant_mytenant | eval keyid=_key
``` in this example, we want to update all entities related to indexes starting by org* ```
| search data_index="org*"
| table keyid
| stats values(keyid) as keys
| eval keys=mvjoin(keys, ",")

Step final: Update entities priority

| inputlookup trackme_dsm_tenant_mytenant | eval keyid=_key
``` in this example, we want to update all entities related to indexes starting by org* ```
| search data_index="org*"
| table keyid
| stats values(keyid) as keys
| eval keys=mvjoin(keys, ",")
``` finally call the endpoint, which supports multiple entities using the Splunk map command```
| map search="| trackme url=/services/trackme/v2/splk_dsm/write/ds_update_priority mode=post body=\"{'tenant_id': 'mytenant', 'priority': 'high', 'keys_list': '$keys$'}\""

Hint

API endpoints per component: (look into menu API & Tooling / TrackMe Rest API Reference for all API endpoints documentation)

  • splk-dsm: /services/trackme/v2/splk_dsm/write/ds_update_priority

  • splk-dhm: /services/trackme/v2/splk_dhm/write/dh_update_priority

  • splk-mhm: /services/trackme/v2/splk_mhm/write/mh_update_priority

  • splk-wlk: /services/trackme/v2/splk_wlk/write/wlk_update_priority

  • splk-flx: /services/trackme/v2/splk_flx/write/flx_update_priority

  • splk-cim: /services/trackme/v2/splk_cim/write/cim_update_priority