The entity state machine¶

Every entity carries a single state — a colour that tells you, at a glance, whether it is healthy. The state is recomputed from scratch on every monitoring cycle, so it always reflects the latest data. This page explains the four states, what moves an entity between them, and the two fields that explain why an entity is where it is.

An entity's overview header showing its state colour and status message

An entity's status message and anomaly reason

The four states¶

State	Score	Meaning
🟢 green	`0`	Healthy. No active anomaly, no override.
🟠 orange	`0 < score < 100`	Warning — a non-critical anomaly is active. Also used as a muted state when an entity is outside its expected monitoring window.
🔴 red	`score ≥ 100`	Critical — the entity has crossed the critical threshold.
🔵 blue	(any)	Suppressed. Not an alarm. An anomaly may be present, but the entity is deliberately muted. Blue always wins over red and orange.

The numeric thresholds (0 / < 100 / ≥ 100) come from the impact score, covered in Impact scoring & thresholds.

What drives a transition¶

Each cycle is idempotent — the same inputs always produce the same state — and runs in three tiers:

Raw state from anomalies and thresholds. No anomalies and all thresholds met → green. Otherwise the impact score decides orange or red.
Logical-group protection. If an entity would be red but belongs to a logical group whose green percentage is still above the group’s minimum, it is demoted to blue — the healthy group carries its struggling member (see Classification & Protection).
Monitoring window and disruption grace. Outside its expected monitoring window the entity is muted to orange; inside a disruption grace period it is held blue so a brand-new anomaly does not alert before it has persisted (see Classification & Protection).

Why an entity is blue¶

Blue is never a failure — it means TrackMe is intentionally holding back. Four mechanisms cause it:

Logical-group protection — the group is mostly healthy.
Disruption grace — the anomaly is too new; TrackMe waits to see if it persists.
Maintenance — the entity, tenant, or instance is in a maintenance window (see Operations & Administration).
Acknowledgment — someone already accepted the problem (see Alerting).

Note

Orange has two meanings. It is both a genuine warning (a non-critical score) and a muted “out of expected window” state. The status message text always tells you which.

Monitored vs unmonitored¶

Independently of its colour, every entity has a monitored state:

enabled — actively monitored and eligible to alert.
disabled — retained but not monitored, and it never alerts.

The main views show enabled entities by default; disabled ones are hidden but still counted — use the table header filter on monitored_state to reveal them. Toggle it per entity, in bulk, or via the API — e.g. | trackme url="/services/trackme/v2/splk_dsm/write/ds_monitoring" mode="post" body="{'tenant_id': 'mytenant', 'action': 'disable', 'object_list': '<entity1>,<entity2>'}" (monitored_state is the persistent KV Store field this sets). It is the right tool when a hybrid tracker discovers entities you do not care about, or when you want to park a noisy entity without deleting it.

Note

Keep three terms distinct: the state (the colour), the monitored state (enabled / disabled), and the score (the number behind the colour).

Status message and anomaly reason¶

Two fields explain an entity’s state:

Status message — a human-readable, colour-matched sentence explaining why the entity has its state. It is the fastest way for an analyst to understand what is wrong, and it appears in the entity’s Status message tab, inside notable events, and in alert results.
Anomaly reason — a short machine code for why the entity is not green, such as delay_threshold_breached, variable_delay_threshold_breached, latency_threshold_breach, outliers_breach, flx_dynamic_threshold_breach:<metric>, wlk_skipped_pct_threshold_breach, or inactivity_threshold_breach. Stateful alerting watches this code and re-opens an alert when the reason changes.

Both are arrays — an entity can be unhealthy for more than one reason at once. Which reasons are possible depends on the component (delay / latency for the feeds family; dynamic thresholds for FLX / FQM / WLK; orphan, version drift, and app-disabled for WLK).

Status message examples¶

A few real status_message / anomaly_reason payloads, as TrackMe stores and surfaces them.

Green — every condition met (here a feed with healthy delay and latency, carried in a logical group):

{
  "status_message": [
    "monitoring conditions for event delay are met. Event delay is 49.565 seconds (duration: 00:00:49), which is lower than the maximum allowed delay of 3600 seconds (duration: 01:00:00), latest event available (_time): 15 Aug 2025 10:55",
    "monitoring conditions for ingest latency are met. Ingestion latency is approximately 21.828 seconds (duration: 00:00:21), which is lower than the maximum allowed latency of 3600 seconds (duration: 01:00:00), latest event indexed (_indextime): 15 Aug 2025 10:55",
    "Logical Group cribl ... is in normal state. The current green percentage of the group is 33.33% which is higher or equal to the minimal green percentage of 5.0%, object_group_members_count=3"
  ],
  "anomaly_reason": ["none"]
}

Red — latency breach (lag_threshold_breached):

{
  "status_message": [
    "Monitoring conditions are not met due to latency issues. Ingestion latency is approximately 353.0 seconds (duration: 00:05:53), which is higher than the maximum allowed latency of 300 seconds (duration: 00:05:00) ... this indicates that the source is receiving delayed events only"
  ],
  "anomaly_reason": ["lag_threshold_breached"]
}

Red — delay breach, static policy (delay_threshold_breached):

{
  "status_message": [
    "Monitoring conditions are not met due to delay issues. Event delay is 17880.563 seconds (duration: 04:58:00), which is higher than the maximum allowed delay of 3600 seconds (duration: 01:00:00) ... the source is receiving events with timestamps older than the threshold defined for this entity."
  ],
  "anomaly_reason": ["delay_threshold_breached"]
}

Red — delay breach, variable policy (variable_delay_threshold_breached) — the message names the active slot and its threshold:

{
  "status_message": [
    "Monitoring conditions are not met due to variable delay threshold breach (active slot: 'business_hours'). Event delay is 7200.0 seconds (duration: 02:00:00), which is higher than the maximum allowed delay of 3600 seconds (duration: 01:00:00) for the active slot ..."
  ],
  "anomaly_reason": ["variable_delay_threshold_breached"]
}