Splunk Feeds Delayed & Inactive Entities (splk-feeds)

Hint

New delayed entities inspector introduced in TrackMe 2.1.10

TrackMe 2.1.10 introduced a new automated process called delayed entities inspector
This new backend process applies to splk-dsm and splk-dhm components, and is automatically created with the Virtual Tenant and component.
This new process ensures to maintain a near up to date view of entities that fall out of the scope of hybrid trackers, due to high delay and/or latency.
It is configurable at the level of the Virtual Tenant, detects when a given entity is no longer covered by the range of an hybrid tracker, and will trigger a target search to refresh the entity metadata.
The delayed entities inspector uses a multi-ranges approach to classify entities based on their current delay, which influences the frequency at which the tracker will attempt to refresh the knowledge of the entity.
This process allows to slightly reduce the risk of false positives, and avoids confusion with out of date delay knowledge, while maintaining scalability and performance.

1. Introduction

TrackMe discovers and maintain Splunk feeds and their resulting entities using one or more primary scheduled logics, called Hybrid Trackers.

Hint

The following documentation describes TrackMe processes regarding the management of entities which are delayed or become inactive if these entities have not sent data anymore for a long period of time.

Trackers have a time range which defines the earliest and latest time of the events that can be processed by the tracker, for instance:

earliest: -4h / indexed earliest: -4h
latest: +4h / indexed latest: +4h

If the entity stops sending data to Splunk, or sends events with a latency that go beyond these limits, at some point the events will become out of the range of trackers, to maintain their state and TrackMe knowledge, the following happens:

Every 5 minutes, the health tracker is executed and runs various maintenance and verification routines for the whole Virtual Tenant.
For splk-dsm and splk-dhm components, the task untracked_entities is executed and detects entities that are no longer covered by any tracker, maintaining the delay metadata knowledge in TrackMe.
In parallel, the delayed entities inspector regularly reviews and performs target searches to refresh the metadata knowledge of entities that are no longer covered by any tracker and covered by the untracked_entities task from the Health Tracker.

1. Date and time of last feed inspection

The date and time of the last inspection is stored as an epoch time format in the field called “tracker_runtime”.

The TrackMe UI makes this information available in a human readable format in the contextual menu, to access to this menu, right click on the entity name within the Tabulator:

3. Health Tracker (untracked_entities)

Hint

Health Tracker and delayed entities

The health tracker includes a task called untracked_entities which is responsible for detecting entities that are no longer covered by any tracker. (out of range)
Entities can be become out of range of trackers if these are suffering from high delay and/or latency, which makes them unreachable by the main trackers.
The health tracker untracker_entities task ensures to maintain and refresh the calculated delay value, according to the current knowledge of the last event received for the entity.

The TrackMe health tracker is created automatically along with the Virtual Tenant, it performs various verifications and is responsible for various things like maintaining the schema version. (upgrade procedures for TrackMe)

It also ensures that inactive entities for the splk-dsm/splk-dhm components are updated regularly, logs for inactive entities updates can be found here:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth task="untracked_entities"

You can include the tenant_id and component if you want to focus on a specific Virtual Tenant and component:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth tenant_id="mytenant" component="splk-dsm" task="untracked_entities"

A typical log activity if there are inactive entities will be similar to:

2023-09-24 08:41:22,647 INFO trackmetrackerhealth.py generate 556 tenant_id="01-feeds", component="splk-dsm", task="untracked_entities", results="{'tenant_id': '01-feeds', 'report_entities_count': '2', 'objects': ['webserver:apache:error', 'webserver:nginx:plus:error']}"

Using this workflow, TrackMe ensures that entities Metadata remain up to date even if they are not covered by any tracker for any reason, such as inactivity (feed interruption) or any other reason.

4. Delayed entities inspector

Hint

Delayed entities inspector

The delayed entities inspector is a new process introduced in TrackMe 2.1.10.
This process performs target searches to refresh the metadata knowledge of entities that are no longer covered by any tracker.
This process is automatically created with the Virtual Tenant and component.

How does it work?

The delayed entities inspector is a scheduled job called:

trackme_<component>_delayed_entities_inspector_tracker_tenant_<tenant_id>

The job calls a custom command trackmesplkfeedsdelayedinspector which performs the following actions:

Retrieves the list of entities that have been managed by the health tracker untracked_entities task, relying on the field tracker_health_runtime
Classify entities based on their delay and according to the Virtual Tenant configuration.
Iterates over entities and performs a target search to refresh the metadata knowledge of the entity.

Virtual Tenant delayed inspector configuration

The delayed inspector configuration is available in the Virtual Tenant configuration page, in the Delayed entities inspector section.

Entities are classified based on these configuration the the following ranges:

24 hours range: If the delay is less than 24 hours, the delayed entities inspector will attempt to refresh the knowledge based on the 24 hours range. (every 30 minutes by default)
7 days ranges: If the delay is less than 7 days, the delayed entities inspector will attempt to refresh the knowledge based on the 7 days range. (every 1 hour by default)
Until disabled: If the delay is greater than 7 days and until the auto disablement period is reached, the delayed entities inspector will attempt to refresh the knowledge of the entity based on the range setting. (once every 24 hours by default)

Execution of the delayed entities inspector searches

The delayed entities inspector iterates over entities and performs a target search to refresh the metadata knowledge of the entity.

Its activity can be tracked through the main logs:

index=_internal sourcetype=trackme:custom_commands:trackmesplkfeedsdelayedinspector

Typically, the logs will show two steps per entity, which notably details the search executed, its runtime and results:

Executing entity delayed tracking resulting search=
delayed_entity_search_results=

You can track executed searches and related events for a given entity by using the following search:

index=_internal sourcetype=trackme:custom_commands:trackmesplkfeedsdelayedinspector tenant_id="mytenant" component="splk-dsm" object="myentity"

Hint

Handlers events introduced in TrackMe 2.1.11

Introduced in TrackMe 2.1.11, the Handler events allows easily tracking the execution of the various TrackMe logics, from hybrid trackers to any meaningful process and notably the delayed entities inspector.
You can easily observe and review the activity of TrackMe such as the delayed entities inspector by looking at the Handler events tab or the trackme:handlers sourcetype in TrackMe’s summary indexes.

An entity affected by high latency and/or delay and becoming out of range, therefore maintained by the delayed entities inspector will show like the following:

From TrackMe 2.1.11, the Handler events will also show the activity associated with the delayed entities inspector:

As soon as the entity issues are fixed, the feed has finished catching up or the underneath issue is resolved, the entity is again managed by the hybrid tracker: