Outlier detection

TrackMe includes a built-in machine-learning engine that learns each entity’s normal numeric behaviour from history and flags deviations. It runs natively inside TrackMe — no Splunk Machine Learning Toolkit dependency — with per-tenant models and impact-based scoring, so an anomaly rarely raises a false alert on its own.

How TrackMe learns and monitors normal behaviour Four stages show historical KPI observations, aggregation and seasonality, baseline learning, and live monitoring. Accepted outliers then contribute a model score to the entity impact score. How TrackMe learns and monitors normal behaviour One model learns one KPI. Seasonality supplies the context; impact-based scoring supplies the operational meaning. 1 — OBSERVE Historical KPI values TrackMe stores measurements for the entity and selected metric. Evidence comes from history A longer window provides more examples of expected behaviour. ENTITY + KPI 2 — PREPARE Aggregate and group Values are reduced per KPI span, then grouped by seasonal context. CALCULATION 10m · avg SEASONALITY EXAMPLE 08:00 09:00 10:00 Compare like with like Monday at 09:00 can learn a different normal than midnight. SPAN + METHOD + TIME 3 — LEARN Fit the normal baseline The native engine fits a statistical distribution for each seasonal group. LOWER UPPER FULL refit the current window INCREMENTAL · NATIVE KV STORE refit a bounded retained sample 4 — MONITOR Evaluate each cycle The current KPI is compared with the bounds for its seasonal group. The model produces evidence Normal, lower breach, or upper breach — plus bounds and reasons. From statistical deviation to operational signal QUALIFY THE BREACH Boundary safety · minimum-value guard confidence gate · model enablement EMIT THE MODEL SCORE Only an accepted breach with normal confidence contributes its configured score. COMBINE ENTITY EVIDENCE The outlier sub-score joins freshness, SLA, quality, and other TrackMe signals. ENTITY state changes at score threshold

The important distinction is between a statistical deviation and an operational alert. A model first decides whether the KPI is outside its expected range. TrackMe then qualifies that evidence and, when accepted, adds the model’s score to the entity’s other signals. The entity changes state only when the combined impact score reaches its configured threshold.

An entity's Outliers anomaly detection tab — the impact-score and lower/upper-bound outlier counters, and the metric-over-time chart with the learned thresholds and flagged anomalies; the status reason shows ML_outliers_detection

The entity’s Outliers anomaly detection tab is where this surfaces: counters for the outlier impact score and lower/upper-bound (and rejected / corrected) outliers, and a chart of the metric over time against the model’s learned thresholds, with anomalies flagged. When an outlier drives the state, the status reason reads ML_outliers_detection.

Note

This page is the overview. For the full reference — the density-function engine, true-context simulation, managing and tuning models, the backend, system-wide options, and troubleshooting — see Outlier detection — in depth. To understand why a model may retain history across training cycles, see Incremental baseline training.

How it works

Detection runs in two phases:

  • Training (trackmefit) — TrackMe builds and trains a model from the entity’s historical metrics and stores it. Models are created automatically when entities are discovered, and retrained on a regular cadence. Full training rebuilds the current calculation window; opt-in incremental training retains a bounded sample across cycles.

  • Monitoring (trackmeapply) — each cycle, the trained model is applied to the current value, producing an is-outlier flag, the expected boundaries, and a confidence level.

The native engine auto-selects the best-fit statistical distribution per model (or you can pin one), and models are stored per tenant for clean lifecycle management and good search-head-cluster performance.

How to read an Outliers result A metric is compared with a learned expected band. A breach is a candidate outlier. Boundary corrections, minimum-value rejection, and confidence determine whether a score is emitted. How to read an Outliers result A bound breach starts a decision, not an automatic alert. TrackMe qualifies the evidence before it can affect entity state. Metric value versus the learned seasonal band The band can change by hour or weekday because each seasonal group is learned independently. UPPER BREACH NORMAL UPPER BOUND LOWER BOUND TIME → inside band: normal outside band: candidate expected range for this context What happens after a breach? 1 · CANDIDATE OUTLIER The current value is outside a learned bound. 2 · BOUNDARY SAFETY Degenerate or implausible bounds are corrected. Visible in the Corrected Lower/Upper counters. 3 · MINIMUM-VALUE GUARD A configured floor can reject the candidate. Rejected candidates are recorded but not scored. 4 · CONFIDENCE GATE Low confidence keeps the result visible, but suppresses scoring until enough history exists. ACCEPTED + NORMAL CONFIDENCE → MODEL SCORE Important: an outlier score is one signal, not the whole decision. TrackMe combines it with every other entity score. State changes only when the total reaches the configured threshold. “Set as false positive” cancels the current outlier contribution without disabling or retraining the model.

A bound breach is therefore a candidate outlier, not an automatic alert. Boundary safety can correct an unusable or implausible range, a configured minimum-value rule can reject the candidate, and low confidence keeps the result visible while suppressing its score. Only an accepted breach with normal confidence contributes the model’s configured score.

Seasonality

Models can learn time-based patterns, so “normal” shifts by time of day or day of week. Choose a time factor:

  • none — for steady, time-independent KPIs (a fixed-rate feed, a constant host count).

  • hour of day — 24 slots.

  • day of week — 7 slots.

  • day-of-week × hour-of-day — for strongly weekly patterns.

Tip

Match the seasonality to the metric. Using hour of day on a metric that really varies by weekday misses the pattern; applying any seasonality to a non-seasonal KPI just adds noise — set it to none.

Confidence and scoring

  • Confidence is low until the model has enough history (by default 30 days) and normal thereafter. A low-confidence model still runs, but its score contribution is suppressed so an undertrained model cannot raise false alerts.

  • A detected outlier contributes an outlier sub-score to the entity’s total score (see Impact scoring & thresholds). By default that contribution sits below the critical threshold, so an outlier alone will not turn an entity red unless you raise its weight or it combines with other anomalies.

Note

Training data. Allow at least the minimum history (30 days by default, governed by splk_outliers_min_days_history) for normal confidence — for seasonal models such as day-of-week × hour-of-day this also gives each slot enough samples.

Handling false positives

The Set as false positive action on an entity’s Outliers tab suppresses an outlier-driven alert without retraining or disabling the model — it cancels the current outlier score, and thanks to the immediate score cache it takes effect on the very next evaluation. It is the manual counterpart to the automatic low-confidence suppression, and works for a single outlier or a whole entity. Per-model threshold guards and deviation checks reject outliers outside sensible value bounds.

You can also simulate a model in a true-to-life preview before applying changes, and run train / monitor / enable / disable / reset in bulk from the Tenant Home. The ML Outliers Anomaly Detection view lists an entity’s models with their status and last train / monitor times, shows the model’s confidence (here normal — enough history to proceed), and runs a simulation over a chosen window so you can see exactly where the model would flag anomalies before committing:

The ML Outliers Anomaly Detection view — the models table (model id, KPI metric, status, last trained, last monitored), a normal-confidence banner, and simulation results charted over the last 7 days with flagged anomalies

Which components benefit

Component

Outlier fit

VOL

The purpose of the component — outliers on the rolling licensed volume of every index (drops and / or spikes) are its state. Always on; one model per index by default, trained on history backfilled at creation. See VOL — Volume Outliers.

DSM, FLX

Natural fit — continuous, often seasonal metrics. Enabled by default.

DHM

Available but selective — per-host volume is noisy. Off by default.

FQM, WLK

Available but not recommended — field-quality and scheduler metrics are too slow-moving, discrete, or regime-dependent. Off by default.

MHM

Not applicable — outlier detection is not available for the Metric Hosts Monitoring component.

Where it applies, outlier detection is enabled or disabled per tenant and per component.

Choosing where behavioural detection runs

“Host outliers or not” is usually the wrong question. Host-level (DHM) outlier detection ships off by default because most environments have limited control over endpoints and are quickly overwhelmed by host count and granularity — but that is a default, not a prohibition, and the value depends entirely on which hosts:

  • Detecting a suspicious change in volume on an Active Directory / domain controller is genuinely valuable.

  • Trying to detect volume variation on an ordinary laptop or workstation feed has little to no value.

The real question is scoping outlier detection to the entities where behavioural change is meaningful. Rather than a blanket on/off, use the Outliers scope, entity labels and tags, and policy management to enable models only for the hosts and feeds that matter — critical servers, sensitive sources — and leave the long tail of low-value endpoints out. That preserves the value while cutting the cost. For broad behavioural detection on volume, the right level is the index, and the right source is the license usage log: that is exactly what the Volume Outliers (VOL) component does — one entity per index, one incremental search per tenant, the history backfilled at creation, drops and spikes per index — available on every edition, Foundation included:

Where to run behavioural outlier detection Three options compared. DHM per host: highest entity count, not the default. DSM per feed: lower granularity, valid with design. VOL on the license usage log: one entity per index, one incremental search, backfilled, recommended. DHM — per host Highest entity count and fan-out. Behavioural scoring is hard to make meaningful at endpoint scale. Off by default DSM — per feed Event counts per index and sourcetype. Valid once you exclude non-meaningful sourcetypes. Valid, with design VOL — per index Licensed volume from the license usage log: one incremental search, backfilled, drops and spikes per index. Recommended Efficiency improves as granularity drops, left to right.

Note

Before 2.4.18 the recommended index-level option was a dedicated tenant built on Flex Objects over the licensing logs (splk_license_usage_per_index_* templates). VOL supersedes it: same source, purpose-built collection and UI, no template to configure, and included in Foundation. The templates remain in the Flex library for custom needs; the Volume Outliers white paper walks the new component end to end.

How it is controlled, tenant-level (applied once):

  • Whole tenant on/off — Configuration → Virtual Tenants → edit the tenant → Enable Machine Learning, under Feature Behavior Settings.

  • Which components run outliers — the Enable Machine Learning allowlist, in the same editor. Dropping DHM from the allowlist is the surgical option: ML stays on for DSM / FLX.

  • Which entities carry models — the ML Outliers management Outliers scope, plus entity labels and tags.

Note

The ML cost lever is scope, not cadence. Do not reduce the ML-monitor frequency to save cost — that opens scoring gaps and delays detection. Choose which entities carry models instead.

See also