Disruption queue

Some entities dip into red for short-lived, self-correcting reasons — a two-minute indexing hiccup, a momentary source outage, a brief drop in host count. Paging on every such blip is pure noise. The disruption queue is the safeguard against those false positives: when an entity first goes red, TrackMe holds it in blue for a grace period and only lets it turn red (and alert) if it stays red continuously beyond that grace. A problem that clears within the window never alerts.

How it works

Each entity has a grace duration. When it first goes red, a timer starts and the entity is shown blue (suppressed). From there:

• if it is still red after the grace duration, it becomes red and can alert;

• if it recovers at any point, the timer resets — the grace measures uninterrupted disruption only, so the next red run starts the clock again.

A sustained problem (grace = 5 min):
  actual:  green ──► RED ───────────────────────────────► RED (still red)
                     │◄────── 5 min grace ──────►│
  shown:   green     │◄─────── BLUE ────────────►│ RED  → alert fires

A short blip (clears inside the grace):
  actual:  green ──► RED ──► green
  shown:   green     BLUE    green                       → no alert, ever

While an entity is in the grace window its status message says exactly that — it has an active anomaly, but the disruption duration has not yet reached the minimal time:

Configuring the grace period

The grace duration cascades — the most specific wins:

1. a system default (typically 5 minutes);

2. a tenant default — set under Configure tenant → Disruption Queue → Default disruption grace period, applied to every entity in the tenant;

3. a per-entity override — set from the entity’s Disruption queue action (0 seconds up to 48 hours).

For the Flex Object (splk-flx) component there is an additional level: a default disruption grace can be set at the Flex Object tracker level, applied to the entities that tracker discovers. Setting the grace to 0 disables it — red is immediate, no grace.

Note

The disruption queue applies to all components and was introduced in TrackMe 2.1.18.

What blue buys you

While the entity is protected, it is blue — and blue is genuinely quiet, not just visually different:

• No alert opens, updates, or closes for it.

• It does not count as red against SLA — a protected entity keeps its compliance and accrues no critical-red time.

• It does not flap red ↔ green; the grace absorbs the churn (see Status flapping).

Note

• The disruption queue is one of the two sources of the blue state; the other is logical-group protection. If both apply, the group is named in the status message.

• It defers the red state — it is not an “I know about this, stop telling me” switch. For that, use acknowledgment (which is independent — an entity can be both acked and in the grace window).

Warning

Tune it to your environment. Too long, and an entity that is genuinely broken can sit in blue and never reach red. Too short, and entities flap blue/red. Match the grace to how long your real, transient blips actually last.

See also

• The entity state machine — where the grace fits in the state machine.

• Status flapping — using the grace window to tame flapping.

• Logical groups — the other source of the blue state.

• Acknowledgments — acknowledgment, the deliberate “silence me” tool.