Impact scoring & thresholds

Behind every state colour is a number: the impact score. It is the most powerful — and the most misunderstood — dial in TrackMe. This page builds the model up step by step: how the score is composed, the crucial difference between problems that score immediately and problems that accumulate over time, and how you can clear or push an entity’s score by hand.

Tip

This page is the conceptual guide. For the exhaustive reference — every weight, every field, the SPL behind it — see the Impact Scoring white paper.

The score in one line

Each cycle, TrackMe sums everything contributing to an entity into a single total_score:

total_score = base_score            # rolling 24h history (the running total of events)
            + impact_score_delay     # while the delay threshold is breached
            + impact_score_latency   # while the latency threshold is breached
            + impact_score_outliers  # the accumulated ML-outlier sub-score
            + threshold_scores       # breached dynamic thresholds (FLX/FQM/WLK)
            + impact_score_<…>       # inactivity, future data, host-count drift, …

…and maps that number to a colour:

• total_score == 0 → 🟢 green

• 0 < total_score < 100 → 🟠 orange

• total_score ≥ 100 → 🔴 red

By convention 100 is the critical line. A weight of 100 means a single breach is enough to turn an entity red; smaller weights mean several problems must add up first. (A separate guard can still override the colour to blue when an entity is suppressed — see The entity state machine.)

The three kinds of contribution

This is the part worth slowing down on, because almost every “why is the score doing that?” question comes back to it. Everything that adds to the score is one of three kinds, and they behave very differently.

                              total_score
                                   │
      ┌────────────────────────────┼────────────────────────────┐
      ▼                            ▼                            ▼
① LIVE breaches            ② SCORE EVENTS               ③ HISTORY
delay · latency ·          outliers · manual ·          base_score
dynamic thresholds ·       false-positive
inactivity · future
      │                            │                            │
re-checked every cycle      discrete events written       running 24-hour
from the current state      to a rolling 24-hour ledger   sum of all events
      │                            │                            │
present only WHILE the      pile up while problems        carries residual
condition is true; gone     recur; each is small;         score between
the instant it clears       they age out after 24h        breaches; fades 24h

1. Live breaches answer “is it broken right now?”. A delay, latency, dynamic-threshold, inactivity, or future-data breach is re-evaluated every cycle. While the condition holds, its weight is added; the moment it clears, the contribution is simply not added on the next cycle — it disappears instantly, with no lag.

2. Score events answer “how many anomalies lately?”. Some signals — above all ML outliers, plus your own manual and false-positive adjustments — are written as individual events into a rolling 24-hour ledger. Each event adds a little, they accumulate while problems recur, and they fall off the back of the window after 24 hours.

3. History (base_score) is the running sum of all those events over the last 24 hours. It is why a long-troubled entity carries weight even between individual breaches, and why it takes a little while to settle fully back to green.

The entity’s Impact Score Definition view shows this directly — the base_score, the score_outliers sub-score, the contributing event sources, and the score over time:

Why outliers don’t behave like a delay breach

The classic surprise. A single delay breach (weight 100) turns an entity red at once, and green again the moment data flows. A single ML outlier is deliberately worth only about 36 — below the red line — so one anomaly leaves the entity orange, not red. Repeated outliers within the 24-hour window add up:

1 outlier   →  ~36   🟠 orange   (a nudge, not an alarm)
2 outliers  →  ~72   🟠 orange
3 outliers  → ~108   🔴 red      (now it's a sustained pattern)
… time passes, events older than 24h drop off → the score eases back down

This is TrackMe’s core false-positive defence: a one-off blip cannot page you; only a recurring pattern escalates. (An undertrained model contributes nothing at all — see Machine Learning.)

Important

It is normal for a fixed problem to stay orange for a while. The live breach clears instantly, but the accumulated outlier events and the base_score history linger until they age out of the 24-hour window. The entity is remembering, not malfunctioning.

Delay and latency

For the feeds family (DSM, DHM, MHM), two live KPIs do most of the work:

KPI	Definition	A breach usually means
Delay	now() − last_event_time — how late the newest data is.	A broken or stalled data flow (process, config, or network failure).
Latency	_indextime − _time — the lag between an event happening and being indexed.	Network pressure, time-zone misconfiguration, or bad timestamp parsing.

The effective delay threshold for an entity is resolved in order — the first that applies wins: a per-entity override (optionally locked so nothing rewrites it); a variable-delay rule that changes by time slot; a lagging class (a named, reusable threshold profile); then the tenant default. Adaptive delay can compute these from history. The full delay configuration lives with the DSM component.

The per-entity lag-monitoring policy is where you see the delay and latency weights, the threshold lock, and the future-tolerance setting together:

Dynamic thresholds (FLX, FQM, WLK)

FLX, FQM, and WLK do not use delay/latency; they evaluate dynamic thresholds on the metrics they emit. A rule is a metric name, an operator (lt/le/gt/ge/eq/ ne), a target value, and its own score. A breach adds that score on top of the component’s threshold-breach weight — these are live contributions, present only while the rule is breached. FLX thresholds can also be time-sliced; WLK supports a tenant-wide default threshold.

Outliers as score events

Each ML model carries its own impact score for a detected outlier (the default is about 36). When the model flags a value, it writes one score event — that is what accumulates into the score_outliers sub-score described above. The model’s impact score, KPI, seasonality and thresholds are all set in its configuration:

Outlier detection, training, confidence, and seasonality are covered in Machine Learning.

Clearing or pushing the score by hand

Because the score is partly a ledger of events, you can write your own events to it — to clear a false alarm immediately, or to force attention. There are three actions, and the right mental model is “I am adding a negative (or positive) score event”, never “I am changing a weight”.

Action	What it does
Set as false positive (entity)	Writes a negative event equal to the entity’s current score, so the total drops to zero on the next cycle and the entity goes green. Clicking again just replaces the previous one — you cannot over-correct. It does not touch the outlier sub-score.
Set as false positive (Outliers tab)	The same idea, but clears only the accumulated outlier score, leaving the rest of the total intact.
Manually influence the score (add / subtract)	Nudge the score up or down by a value you choose. Manual adjustments accumulate. Pushing the score up without a real anomaly records a score_breached anomaly so the change is visible and audited.

Important

These adjustments take effect on the next cycle (an immediate cache means you do not wait for indexing) and they fade after 24 hours. They suppress for the window — they do not permanently change the entity. To stop outlier alerts for good, fix or retrain the model, do not keep marking false positives.

Warning

Never try to cancel a condition with a negative weight. To reduce score, lower the relevant weight, or use the false-positive / manual-decrease actions above.

Tuning the weights

Every contribution above has a weight you can change. Weights cascade — later wins:

1. a built-in fallback;

2. the system default (affects new tenants only);

3. per tenant (affects every entity in that tenant);

4. per entity (overrides the tenant value for one entity).

Setting a weight to 0 disables that contribution for the chosen scope. The weights are set at tenant creation and from the per-tenant Impact Score Configuration:

Note

The defaults are deliberately not all 100. Typically a delay breach and a host-count breach weigh 100 (red on their own), while latency weighs around 48 and outliers, data-sampling, and future-tolerance weigh around 36 — so a latency breach alone, or a single outlier, lands at orange, not red. That is the false-positive tuning at work, not a bug. The exact values live in your tenant settings; tune from there.

See also

• Impact Scoring white paper — the full reference.

• The entity state machine — how the score becomes a colour.

• Machine Learning — outlier detection, training, and false-positive handling.

• DSM — Data Source Monitoring — delay and latency configuration in depth.