Status flapping

An entity that rapidly flips between healthy and unhealthy — flapping — is telling you something different from one that is steadily broken. TrackMe records every state transition so you can spot, and tame, unstable entities.

What flapping is

Every time an entity changes state, TrackMe writes a flipping event recording the previous state, the new state, the anomaly reason, and the time. The first event for any entity is its discovery — a flip from the synthetic discovered status to its first computed state. For example:

object=… flipped from previous_state=discovered to state=green  anomaly_reason=none
object=… flipped from previous_state=green      to state=red    anomaly_reason=delay_threshold_breached
object=… flipped from previous_state=green      to state=red    anomaly_reason=variable_delay_threshold_breached
object=… flipped from previous_state=red        to state=green  anomaly_reason=none

Flapping is tracked for the splk-dsm, splk-dhm, splk-mhm, splk-cim and splk-flx components.

Why it matters

The frequency of flips is itself a signal. A steadily-red entity is a known problem; an entity bouncing red ↔ green every few minutes usually means a threshold set too tight for a bursty feed, not a real outage. The repeated anomaly_reason on each flip tells you which condition is responsible — a red → green → red pattern with the same reason is the classic “threshold too tight” signature.

Where to review it

  • Open the entity’s Status Flipping tab — an over-time chart of flip frequency plus a table ordered by the most recent transitions.

  • Or search directly in Splunk, for example:

    `trackme_idx(mytenant)` sourcetype=trackme:state tenant_id="mytenant" object="myobject"
    

Flipping data also feeds TrackMe’s SmartStatus assessment of an entity.

Taming flapping

Flapping is a symptom, not a separate alarm — the fix is to absorb the transient blips:

  • Widen the disruption grace window so a brand-new anomaly is held blue until it has persisted, instead of immediately flipping to red (see Classification & Protection).

  • Tune the threshold that keeps tripping — for a bursty feed, a variable-delay rule or a more appropriate lagging class is usually the answer (see Impact scoring & thresholds).

  • Remember that logical-group protection already dampens flapping: a member of a healthy group shows as a stable blue rather than flipping.

See also