Delay & latency thresholds¶

This is the starting point for splk-feeds thresholds: how the two KPIs work, how to read an entity’s current thresholds, how to set them by hand, and how to validate a value before you commit it. For the automated and rule-based options, see lagging classes, adaptive delay, and variable delay.

Delay and latency, briefly¶

TrackMe measures two independent things for every feed entity, and gives each its own threshold.

Delay — is the data fresh?: How old the most recent event is: delay = now() − max(_time), where _time is when the event happened at the source. It answers “how long ago did the most recent event arrive?” High delay means the source has stopped producing fresh data — it is silent, or only sending old, backlogged events.
Latency — how slow is the pipeline?: How long an event took to travel from the source to Splunk’s index: latency = _indextime − _time, where _indextime is when Splunk indexed it. It answers “how delayed was the pipeline between the source and Splunk?” High latency points to a forwarding, buffering, or queuing problem — the events exist, but they arrived late.

The two are independent — you can have one without the other, and each tells a different story:

Pattern	What it means
High latency, low delay	Events are arriving, but they were already old when they got there — the source is fine, the transport is slow (pipeline buffering).
High delay, low latency	Events are indexed promptly once they arrive, but the source has stopped sending fresh data (a gap or silence).
Both high	The source is sending delayed, backlogged events that also took a long time to transit — the most concerning pattern.

Because they mean different things, tune them independently. On every monitoring cycle TrackMe checks each against its own threshold — delay against the entity’s delay policy (static or variable), latency against data_max_lag_allowed — and each breach contributes to the impact score that drives the entity’s state. A new entity starts with a 1-hour (3600-second) maximum for both; the rest of this page is about setting better values.

Reviewing current thresholds¶

An entity’s active thresholds — and which delay policy is in force (static or variable) — are shown in the entities table (a two-value cell for delay and latency) and on the entity’s own screen.

Setting a threshold per entity¶

To set thresholds by hand, open the entity, go to its threshold settings, and you can:

set the maximum acceptable latency;
set the maximum acceptable delay;
choose whether to alert on both KPIs (the default) or only one;
lock the threshold so neither a lagging class nor adaptive delay will overwrite your value (see Who manages a threshold: the Threshold Lock and the AI Advisor).

Tip

The Threshold Lock replaces the old per-entity “override lagging classes” checkbox. Lock a threshold when you have deliberately tuned it and want it left alone; leave it unlocked to let lagging classes or adaptive delay manage it.

Simulating a threshold before you apply it¶

Before committing a value, use Simulate thresholds to test it against the history TrackMe has already accumulated. Provide candidate latency and delay values and TrackMe replays them over the entity’s summary events, returning one event per breach that would have occurred — so you can see whether a value is too tight before it pages anyone. For a provider that sends data once per hour but indexes it promptly, you would keep latency tight and allow a little over an hour of delay, confirm the simulation is quiet, then apply.

Worked example: an entity suffering from latency¶

Latency means events are indexed well after their event time — you may also see delay, but not necessarily; a feed can mix near-real-time events with late ones. The two measures will usually differ, which is expected.

When an entity breaches its latency threshold, the rising latency is visible in both the overview and Performance Metrics charts, the status message spells out the problem, and Smart Status adds an automated investigation. To review the offending events by hand — here, events indexed more than 15 minutes (900 s) late over the last 4 hours:

index=myindex sourcetype=mysourcetype _index_earliest="-4h" _index_latest="+4h" earliest="-4h" latest="+4h"
| eval indextime=_indextime, latency=(_indextime-_time)
| where latency>900
| eval indextime=strftime(indextime, "%c")
| table _time indextime latency _raw

Once the root cause is fixed and latency returns to normal, TrackMe brings the entity back to green after a short while.

Worked example: delay without latency¶

Some feeds are perfectly on time when they do arrive, but arrive infrequently — batched data. A provider that emits one batch per day shows no latency but accrues delay between batches; TrackMe surfaces the current delay alongside the latest event (_time) and the latest ingested event (_indextime). The fix is to set a delay of a little over 24 hours while keeping latency low, after which the entity returns to green.

A feed that arrives once per hour is the more common case: the default 3600-second delay will alert if the provider runs even slightly late, and too tight a value (say 3000 s) produces constant delay alerts — so you settle on something like 4200 s (1 hour 10 minutes). As always, tolerating delay in delivery does not mean tolerating latency in indexing — tune the two KPIs independently.