Delay vs latency: what they actually mean

People call everything “latency” — a feed is late, a dashboard is stale, an alert fired, and it all gets labelled latency. In TrackMe these are two different measurements that answer two different questions, and telling them apart is the single most useful thing you can learn about feeds monitoring. Get this right and the entities table, the thresholds, and the alerts all suddenly make sense.

Two timestamps

Every event in Splunk carries two moments in time:

  • _time — when the event happened (the timestamp inside the event itself).

  • _indextime — when Splunk indexed it (when it actually arrived and was stored).

Those two are never exactly equal, and the whole idea of delay and latency falls out of the gap between them and the gap to now:

  • Latency = _indextime − _time — “when an event arrives, how late is it?” A measure of the ingestion pipeline (forwarding, queues, network, clock skew). It is a per-event property.

  • Delay = now − last_event_time — “how long since we last heard anything at all?” A measure of freshness: is the feed still alive right now?

Latency and delay on a timeline Latency is the gap between an event's own timestamp (_time) and when Splunk indexed it (_indextime). Delay is the gap between the most recent event and now. Latency — a property of each event latency = _indextime − _time event happens _time indexed by Splunk _indextime Delay — a property of the feed right now delay = now − last event time most recent event _time now

The mailbox analogy

Think of a feed as your postbox:

  • Latency is how long a letter takes from the day it was written (the postmark — _time) to the day it lands in your box (_indextime). A letter postmarked Monday that arrives Thursday has three days of latency.

  • Delay is how long since the last letter of any kind turned up. If nothing has arrived for five days, your delay is five days — the box has gone quiet.

The two are independent. Someone can drop a stack of old, week-old postmarked letters in today (low delay, high latency — fresh arrivals, but every one is stale). Or your reliable daily letter can simply stop coming (high delay, low latency — when letters did arrive they were same-day, but now there is nothing).

Why you can’t collapse them into one

Because they are independent, a feed sits in one of four states — and the cause you go looking for is different in each:

The four combinations of delay and latency A 2x2 grid. Vertical axis is delay (low at bottom, high at top); horizontal axis is latency (low at left, high at right). Bottom-left: healthy. Bottom-right: arriving but time-shifted. Top-left: feed went quiet. Top-right: late and slow. Feed went quiet stopped or paused; when it did arrive it was prompt Late and slow an outage, or a backlog still catching up Healthy data is arriving, and on time Arriving, but time-shifted batch of old data, clock skew, or a slow forwarder Latency → low high Delay → low high

The practical upshot: an alert on latency and an alert on delay are telling you different things. A latency breach points at your ingestion path; a delay breach points at the source (has it stopped?). Fixing one does nothing for the other.

How TrackMe measures each

  • Latency is computed per event as _indextime − _time. On an entity, TrackMe surfaces the current, average, and 95th-percentile latency over a recent window (so a single odd event does not dominate).

  • Delay is computed as now − last_event_time — always relative to this moment, which is why a delay figure keeps climbing on its own while a feed is silent.

Each has its own threshold, and an entity turns red when it goes over:

  • data_max_delay_allowed — the delay max (freshness ceiling).

  • data_max_lag_allowed — the lag max (per-event latency ceiling).

New entities start with a 1-hour default for both. See Feeds thresholds (delay & latency) for the ways to set the right value per feed (static, lagging classes, adaptive, and variable).

Where you see them in the UI

In the entities table, each feed row shows:

Column

Meaning

lag (event / ingest)

A combined summary cell: the current delay (event) and latency (ingest), in that order.

Latest event

The newest event seen, by _time (drives delay).

Latest ingest

The newest event seen, by _indextime.

Delay max

The current delay threshold for the entity (data_max_delay_allowed).

Lag max

The current latency threshold for the entity (data_max_lag_allowed).

On an entity’s own view, the KPI stat tiles read:

  • CURRENT EVENT DELAY — now − last_event_time.

  • AVG INGESTION LATENCY / PERC95 INGESTION LATENCY — the mean and 95th-percentile latency over the recent window.

All of these display in seconds, switching to a [D+]HH:MM:SS duration format (via Splunk’s tostring(…,"duration")) once the value exceeds 60 seconds — so 4200 shows as 01:10:00 and a two-day-stale feed as 2+00:00:00.

See also