Outlier detection¶

TrackMe includes a built-in machine-learning engine that learns each entity’s normal numeric behaviour from history and flags deviations. It runs natively inside TrackMe — no Splunk Machine Learning Toolkit dependency — with per-tenant models and impact-based scoring, so an anomaly rarely raises a false alert on its own.

How TrackMe observes historical KPI values, prepares seasonal groups, learns a full or incremental baseline, monitors new values, qualifies breaches, and contributes accepted model scores to entity state

The important distinction is between a statistical deviation and an operational alert. A model first decides whether the KPI is outside its expected range. TrackMe then qualifies that evidence and, when accepted, adds the model’s score to the entity’s other signals. The entity changes state only when the combined impact score reaches its configured threshold.

An entity's Outliers anomaly detection tab — the impact-score and lower/upper-bound outlier counters, and the metric-over-time chart with the learned thresholds and flagged anomalies; the status reason shows ML_outliers_detection

The entity’s Outliers anomaly detection tab is where this surfaces: counters for the outlier impact score and lower/upper-bound (and rejected / corrected) outliers, and a chart of the metric over time against the model’s learned thresholds, with anomalies flagged. When an outlier drives the state, the status reason reads ML_outliers_detection.

Note

This page is the overview. For the full reference — the density-function engine, true-context simulation, managing and tuning models, the backend, system-wide options, and troubleshooting — see Outlier detection — in depth. To understand why a model may retain history across training cycles, see Incremental baseline training.

How it works¶

Detection runs in two phases:

Training (trackmefit) — TrackMe builds and trains a model from the entity’s historical metrics and stores it. Models are created automatically when entities are discovered, and retrained on a regular cadence. Full training rebuilds the current calculation window; opt-in incremental training retains a bounded sample across cycles.
Monitoring (trackmeapply) — each cycle, the trained model is applied to the current value, producing an is-outlier flag, the expected boundaries, and a confidence level.

The native engine auto-selects the best-fit statistical distribution per model (or you can pin one), and models are stored per tenant for clean lifecycle management and good search-head-cluster performance.

How to interpret a TrackMe Outliers result, from a value outside the learned seasonal band through boundary safety, minimum-value rejection, confidence gating, and accepted model scoring

A bound breach is therefore a candidate outlier, not an automatic alert. Boundary safety can correct an unusable or implausible range, a configured minimum-value rule can reject the candidate, and low confidence keeps the result visible while suppressing its score. Only an accepted breach with normal confidence contributes the model’s configured score.

Seasonality¶

Models can learn time-based patterns, so “normal” shifts by time of day or day of week. Choose a time factor:

none — for steady, time-independent KPIs (a fixed-rate feed, a constant host count).
hour of day — 24 slots.
day of week — 7 slots.
day-of-week × hour-of-day — for strongly weekly patterns.

Tip

Match the seasonality to the metric. Using hour of day on a metric that really varies by weekday misses the pattern; applying any seasonality to a non-seasonal KPI just adds noise — set it to none.

Confidence and scoring¶

Confidence is low until the model has enough history (by default 30 days) and normal thereafter. A low-confidence model still runs, but its score contribution is suppressed so an undertrained model cannot raise false alerts.
A detected outlier contributes an outlier sub-score to the entity’s total score (see Impact scoring & thresholds). By default that contribution sits below the critical threshold, so an outlier alone will not turn an entity red unless you raise its weight or it combines with other anomalies.

Note

Training data. Allow at least the minimum history (30 days by default, governed by splk_outliers_min_days_history) for normal confidence — for seasonal models such as day-of-week × hour-of-day this also gives each slot enough samples.

Handling false positives¶

The Set as false positive action on an entity’s Outliers tab suppresses an outlier-driven alert without retraining or disabling the model — it cancels the current outlier score, and thanks to the immediate score cache it takes effect on the very next evaluation. It is the manual counterpart to the automatic low-confidence suppression, and works for a single outlier or a whole entity. Per-model threshold guards and deviation checks reject outliers outside sensible value bounds.

You can also simulate a model in a true-to-life preview before applying changes, and run train / monitor / enable / disable / reset in bulk from the Tenant Home. The ML Outliers Anomaly Detection view lists an entity’s models with their status and last train / monitor times, shows the model’s confidence (here normal — enough history to proceed), and runs a simulation over a chosen window so you can see exactly where the model would flag anomalies before committing:

The ML Outliers Anomaly Detection view — the models table (model id, KPI metric, status, last trained, last monitored), a normal-confidence banner, and simulation results charted over the last 7 days with flagged anomalies

Which components benefit¶

Component	Outlier fit
DSM, FLX	Natural fit — continuous, often seasonal metrics. Enabled by default.
DHM	Available but selective — per-host volume is noisy. Off by default.
FQM, WLK	Available but not recommended — field-quality and scheduler metrics are too slow-moving, discrete, or regime-dependent. Off by default.
MHM	Not applicable — outlier detection is not available for the Metric Hosts Monitoring component.

Where it applies, outlier detection is enabled or disabled per tenant and per component.

Choosing where behavioural detection runs¶

“Host outliers or not” is usually the wrong question. Host-level (DHM) outlier detection ships off by default because most environments have limited control over endpoints and are quickly overwhelmed by host count and granularity — but that is a default, not a prohibition, and the value depends entirely on which hosts:

Detecting a suspicious change in volume on an Active Directory / domain controller is genuinely valuable.
Trying to detect volume variation on an ordinary laptop or workstation feed has little to no value.

The real question is scoping outlier detection to the entities where behavioural change is meaningful. Rather than a blanket on/off, use the Outliers scope, entity labels and tags, and policy management to enable models only for the hosts and feeds that matter — critical servers, sensitive sources — and leave the long tail of low-value endpoints out. That preserves the value while cutting the cost. For broader behavioural detection, run it at the index level (DSM), or in a dedicated tenant built on Flex Objects over the Splunk licensing logs (pre-built use cases ship for this):

How it is controlled, tenant-level (applied once):

Whole tenant on/off — Configuration → Virtual Tenants → edit the tenant → Enable Machine Learning, under Feature Behavior Settings.
Which components run outliers — the Enable Machine Learning allowlist, in the same editor. Dropping DHM from the allowlist is the surgical option: ML stays on for DSM / FLX.
Which entities carry models — the ML Outliers management Outliers scope, plus entity labels and tags.

Note

The ML cost lever is scope, not cadence. Do not reduce the ML-monitor frequency to save cost — that opens scoring gaps and delays detection. Choose which entities carry models instead.