Adaptive delay (ML)¶

Adaptive delay lets TrackMe learn each feed’s normal lag from history and maintain its delay threshold automatically — so a feed whose natural cadence drifts over time keeps a sensible threshold without anyone re-tuning it. It is handled by a dedicated tracker, trackmesplkadaptivedelay, which inspects entities in a delay breach and, when the conditions are met, updates the delay threshold and records an audit trail.

The more history TrackMe has for an entity, the more accurate the computed threshold.

Important

When the AI Feed Lifecycle Advisor covers DSM/DHM, it owns delay management and adaptive delay is disabled automatically (the tracker short-circuits and its tenant toggle is forced off). See Who manages a threshold: the Threshold Lock and the AI Advisor.

How a run works¶

The adaptive-delay tracker runs every 20 minutes by default, as a per-entity batch under a runtime budget (900 seconds by default, never more than the schedule interval — see batches under a runtime budget). Each occurrence selects the entities that need review, processes as many as the budget allows, and resumes with the rest on the following cycle — so on a large tenant a multi-minute run simply means a large batch, not a stuck search.

Adaptive delay is also self-limiting by design: an entity only enters the review bucket when it is actually in a delay-breaching state, and it is further gated by the entity’s SLA compliance — entities comfortably within SLA are skipped. The cost therefore scales with how much of the estate is unstable: the cleaner and more stable the environment, the fewer entities adaptive delay has to review.

How an entity is selected¶

Each run, an entity becomes a candidate only if all of these hold — otherwise it is skipped:

enabled  and  currently red
  └─ in a delay breach (delay_threshold_breached)
       └─ threshold UNLOCKED                  ── locked? → skip (your pinned value stands)
            └─ current delay ≥ min_delay_sec  (3600 s by default)
                 └─ AI Feed Lifecycle Advisor not covering DSM/DHM
                                              ── advisor covers? → skip (it owns delay)
                      └─ ✓ candidate → compute & update the threshold

(See Who manages a threshold: the Threshold Lock and the AI Advisor for the lock and the AI Advisor gate.)

It also reviews entities it has already managed: an entity updated in the last 4 hours is left to settle; one updated 4–24 hours ago whose threshold was increased is re-checked; and beyond that, managed entities are reviewed on a rolling 7-, 15- or 30-day cadence (review_period_no_days) so a feed that has stabilised can have its threshold tightened back down.

How the threshold is computed¶

For a selected entity, TrackMe models the observed lag metric (trackme.splk.feeds.lag_event_sec) with its native ML engine (trackmefit — no Splunk MLTK dependency) to find an upper bound on normal lag, then rounds to the nearest hour and adds a one-hour safety buffer (the variable-delay path rounds up instead). In essence:

| mstats latest(trackme.splk.feeds.lag_event_sec) as lag_event_sec
    where `trackme_metrics_idx(mytenant)` tenant_id="mytenant"
    object_category="splk-dsm" object="myobject" by object span=5m
| trackmefit ...                      ``` native ML upper-bound on normal lag ```
| eval adaptive_delay = (round(UpperBound/3600, 0) * 3600) + 3600

When re-reviewing an entity after an earlier update, TrackMe runs the same logic over three windows — last 24 hours, 7 days and 30 days — and aggregates them, which lets it react faster when a feed returns to stability after an interruption rather than waiting for the long window to catch up. Every search and its result are logged.

Tracker arguments¶

The behaviour is tuned with per-tracker arguments (edit them in the tracker configuration):

Argument	Default	Effect
`min_delay_sec`	`3600`	Minimum current delay (seconds) for an entity to be considered.
`max_auto_delay_sec`	`604800`	Cap on the value the engine may set; a higher computed value is clamped to this (the entity then stays under review).
`max_changes_past_7days`	`10`	Maximum updates per entity in a rolling 7 days; once reached, the entity is left alone until the counter resets.
`min_historical_metrics_days`	`7`	Minimum days of accumulated metrics before the threshold may be updated.
`review_period_no_days`	`30`	Re-review window for already-managed entities (`7`, `15` or `30`).
`max_sla_percentage`	`100`	Skip entities whose current SLA % is at or above this — protecting a stable entity hit by a genuine one-off anomaly from being widened (defaults effectively to 90%).

Automatic updates and audit¶

When the command updates a threshold it writes an audit record (context: automated adaptive delay update). Review them with:

`trackme_audit_idx` tenant_id=* "automated adaptive delay update"
| table _time, tenant_id, object_category, object, action, comment
| sort - 0 _time | trackmeprettyjson fields=comment

A dedicated dashboard, TrackMe - Adaptive delay threshold audit, is available under Audit & Troubleshoot.

Automated summary notes¶

When the framework updates an entity’s threshold, it also attaches a Markdown summary note to that entity (author trackmesplkadaptivedelay), making the change self-documenting. The note records the change direction, the previous and new values (as a duration and in seconds, e.g. 1h (3600s) → 4h (14400s)), a short reason, and a reasoning table (95th-percentile, max and standard deviation of the observed lag plus the ML upper bound, per window), and flags when the value was capped at max_auto_delay_sec.

Find it under Tenant Home → the entity → Notes, or in the entity’s Audit changes tab. Notes are controlled per tenant by the adaptive_delay_notes setting (Feature Behavior, enabled by default); a note is created only when a threshold actually changes, and note creation never blocks the update itself.

Activity logs¶

The tracker logs every entity it considered and the action taken:

index=_internal sourcetype=trackme:custom_commands:trackmesplkadaptivedelay object="myobject"

Opting an entity or tenant out¶

To stop adaptive delay managing a single entity, lock its threshold (see Who manages a threshold: the Threshold Lock and the AI Advisor) — the lock pins your value and excludes the entity from automatic management. To disable adaptive delay for a whole tenant, turn off the adaptive_delay setting under Configure → Virtual Tenants preferences.

Disabling it at the tenant level also takes the backend off the Splunk scheduler: the tenant’s health tracker un-schedules the adaptive-delay tracker on its next cycle (and the tracker short-circuits at runtime in any case). The only effect is that thresholds are no longer auto-reviewed — they simply hold their current values, or whatever your policies and manual settings define. Adaptive delay is a convenience feature, valuable but entirely optional, which makes it the classic first lever in a footprint review.

Variable adaptive delay¶

Adaptive delay is no longer limited to the static policy — for an entity on the variable delay policy, the same trackmesplkadaptivedelay tracker (same adaptive_delay tenant setting, same per-entity management) also keeps the slot thresholds current. It recomputes each slot’s max_delay_allowed and the fallback variable_delay_default from history, while preserving your slot layout — the slot names, days and hours you defined are kept; only the threshold values are refreshed.

This is different from the variable-delay auto-review reviewer (trackmesplkvariabledelayreview), which regenerates the slot layout. The two are independently opt-in and can both apply to a variable entity:

Mechanism	Opt-in	What it changes
Variable adaptive delay	`adaptive_delay` + entity managed (unlocked)	Refreshes per-slot thresholds + the default; keeps your slot names/days/hours.
Variable-delay auto-review	`variable_delay_auto_review_enabled`	Regenerates the slot layout (`auto_slot_N`) when patterns shift.

A locked entity (Threshold Lock) is skipped by both, and its pinned slot schedule is restored if it ever drifts.