Splunk Feeds Delayed & Inactive Entities (splk-feeds)

Note

Decomission of the Delayed entities tracker in TrackMe 2.0.64

  • In TrackMe 2.0.64, we have decomissioned the delayed entities tracker, which was automatically deleted by the schema migration once you upgraded to this version

  • This tracker was reponsible for maintaining entities which were not updated by the main trackers for a certain amount of time

  • We have replaced this workflow with a more consistent approach using the health tracker, this ensures a faster and more reliable management of entities that are not covered by the time period of any tracker

  • This processes is fully automated, and there are no actions required

1. Introduction

TrackMe discovers and maintain Splunk feeds and their resulting entities using one or more scheduled logics, called Trackers.

Hint

The following documentation describes TrackMe processes regarding the management of entities which are delayed or become inactive if these entities have not sent data anymore for a long period of time.

Trackers have a time range beginning, and a time range end, for instance:

  • earliest: -4h / latest: -4h

If the entity stops sending data to Splunk, at some point the events will become out of the range of trackers, to maintain their state and TrackMe knowledge, the following happens:

  • Every 5 minutes, the health tracker is executed and runs various maintenance and verification routines for the whole Virtual Tenant

  • Regarding splk-dsm amd splk-dhm components, it will ensure that Metadata are fast refreshed for that any entities that have not been updated since more than 15 minutes

2. Date and time of last feed inspection

The date and time of the last inspection is stored as an epoch time format in the field called “tracker_runtime”.

The TrackMe UI makes this information available in a human readable format in the contextual menu, to access to this menu, right click on the entity name within the Tabulator:

screen1.png

3. Health Tracker (context=”untracked_entities”)

The TrackMe health tracker is created automatically along with the Virtual Tenant, it performs various verifications and is responsible for various things like maintaining the schema version. (upgrade procedures for TrackMe)

health_tracker.png

It also ensures that inactive entities for the splk-dsm/splk-dhm components are updated regularly, logs for inactive entities updates can be found here:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth context="untracked_entities"

You can include the tenant_id and component if you want to focus on a specific Virtual Tenant and component:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth tenant_id="mytenant" component="splk-dsm" context="untracked_entities"

A typical log activity if there are inactive entities will be similar to:

2023-09-24 08:41:22,647 INFO trackmetrackerhealth.py generate 556 tenant_id="01-feeds", component="splk-dsm", context="untracked_entities", results="{'tenant_id': '01-feeds', 'report_entities_count': '2', 'objects': ['webserver:apache:error', 'webserver:nginx:plus:error']}"

Using this workflow, TrackMe ensures entities Metadata remain up to date even if they are not covered by any tracker for any reason, such as inactivity (feed interruption) or any other reason.