Variable delay¶

Variable delay lets an entity’s delay threshold change by day of the week and hour of the day, instead of using one value around the clock. It solves a common problem: feeds have natural rhythms — a business application produces data through weekday office hours but little overnight or at weekends. A single static threshold must be loose enough for the quiet periods, which makes it too forgiving when the feed is busy. Variable delay fixes this by letting you define named time slots, each with its own threshold.

Variable delay applies to splk-dsm and splk-dhm.

Static vs variable delay¶

Aspect	Static (default)	Variable
Threshold	One fixed value at all times	Multiple values that change by day-of-week and hour-of-day
Configuration	A single number per entity	Named slots (days, hours, threshold) + a fallback default
Quiet periods	Must keep the value loose, reducing sensitivity when busy	Relax overnight/weekends, tighten during business hours
Anomaly reason	`delay_threshold_breached`	`variable_delay_threshold_breached`
Entity field	`variable_delay_policy="static"`	`variable_delay_policy="variable"`

Setting the tenant default¶

The default delay policy for new entities is set per tenant, separately for DSM and DHM — both at tenant creation (static, or variable with slot definitions) and afterwards from tenant management.

Setting variable delay on an entity¶

Per entity, variable delay is configured from the entity’s lagging policy / variable-delay modal.

How slots work¶

Each slot specifies a name, the days it covers (0 = Monday … 6 = Sunday), the hours it covers (0–23), and the max delay allowed (seconds). Evaluation is first match wins: slots are checked in order, the first whose day and hour match the current time is used, and if none match, the fallback default threshold applies.

{
  "slots": [
    {
      "slot_name": "business_hours",
      "days": [0, 1, 2, 3, 4],
      "hours": [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
      "max_delay_allowed": 3600
    },
    {
      "slot_name": "weekends",
      "days": [5, 6],
      "hours": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23],
      "max_delay_allowed": 86400
    }
  ]
}

Here weekday business hours allow 1 hour of delay, weekends allow 24 hours, and weekday nights fall through to the default. The UI ships presets — business hours, weekday/weekend, and a three-tier business hours / nights / weekends — for quick setup.

Auto-computation from history¶

TrackMe can propose slots from an entity’s own history, using trackme.splk.feeds.lag_event_sec: it buckets 5-minute metrics by day-of-week × hour-of-day, applies a statistical function (perc95 by default, or perc99), rounds each window up to the next hour plus a one-hour buffer (minimum 3600 s), groups adjacent hours with the same value into named slots, and applies a safety cap (default 604800 s = 7 days).

Auto-review¶

An optional review process keeps slots current as patterns shift. A scheduled report per tenant per component (DSM, DHM) runs daily — trackmesplkvariabledelayreview — recomputing thresholds from history and updating slots only when the largest deviation exceeds a configurable threshold (default 20%). It can be enabled at the tenant level and per entity.

index=_internal sourcetype=trackme:custom_commands:trackmesplkvariabledelayreview

Note

The periodic reviewer regenerates slot layouts (auto_slot_N); the operator “honour” strategy and the adaptive-delay branch preserve your slot names.

Important

When the AI Feed Lifecycle Advisor covers DSM/DHM, variable-delay auto-review is disabled automatically (and does not auto-re-enable when the advisor is turned off). See Who manages a threshold: the Threshold Lock and the AI Advisor.

Interactions with other features¶

Adaptive delay. Variable delay and adaptive delay are no longer mutually exclusive — adaptive delay also keeps variable-delay slot thresholds refreshed. Whether either mechanism may touch a given entity is governed by the Threshold Lock.
Lagging classes. A lagging class can itself carry a variable-delay schedule; and a class’s latency override still applies independently of the delay policy.
Monitoring time policy. Independent — the monitoring policy controls whether an entity is monitored; variable delay controls which threshold applies when it is.
Bulk edit. Variable delay can be applied to many entities at once from the bulk-edit actions.
Upgrades. Existing entities default to variable_delay_policy="static" with no variable-delay record, so behaviour is unchanged until you opt in.