Volume Outliers¶
Licensed volume per index as entities: rolling volume, daily licensed volume verified against the license usage log, daily changes and the month projection, ML outliers on drops and spikes, and the ML Advisor.
about 35 minutes · 43 slides. Each slide is followed by a description of what it shows — read top to bottom, this page is the walkthrough.
Overview¶
Volume Outliers (VOL) is the seventh TrackMe component, introduced in 2.4.18: the standardised, first-class form of the volume detection TrackMe has always offered. One tenant watches every index of a deployment for abnormal drops, spikes and silence, backfills its history at creation, and adds a cost trend, a month projection and the whole environment’s license usage. Parts 1 to 3 were captured on TrackMe 2.4.18, from an empty tenant on a Splunk Enterprise deployment with 43 indexes in one license pool; the license usage view of Part 4 on TrackMe 2.4.19.
The path runs from the tenant creation and the backfill, through reading the results and tuning the detection, to the global license usage view from scopes to chargeback, alerting, the AI Assistant and an AI ML Advisor review of a volume model.
Parts 2 and 4 were refreshed on TrackMe 2.4.22: binary units with IEC labels, a collection that waits for a late source, the closed-day reconciliation against the license usage log, and the Daily changes and Investigate an index views.
Why volume needs its own signal¶
Availability monitoring answers whether data arrives on time, not whether the amount is normal. A broken forwarder, a lost route or a parser change leaves the feed green while half of the data is missing; a debug level left on is an overage at month end; a silent index still exists while its last event is a week old.
VOL watches the volume itself, from the license usage log Splunk bills, read incrementally in 5-minute buckets. Every index becomes a TrackMe entity with an ML outliers model on its rolling volume, an inactivity policy, the daily licensed volume, a 30-day trend and a month projection, on Enterprise or Cloud, in the Foundation edition.
Source, tracker, KPIs, detection, action¶
Callouts 1 to 5 follow the pipeline. The source is the license usage log: what Splunk bills, one log for every index, on Enterprise and on Cloud. One incremental tracker collects every 5 minutes, only the new events, into 5-minute buckets per index, never rescanning. The KPIs are per-index trends: rolling 60 min, 4 h, 12 h and 24 h volume and the daily licensed volume, as trackme.splk.vol.* metrics.
Detection runs one ML model per index on the rolling volume, drops and / or spikes, and a silent index turns red. Action is the impact score to state, stateful alerts, the 30-day trend and the month projection. The same diagram sits under How it works in the wizard.
The four parts¶
One Virtual Tenant covers the whole deployment. Part 1, Setup, creates the tenant through the wizard (basics, volume, inactivity, RBAC), replays 30 days of history at creation, and follows day one from pending to complete. Part 2, Read, covers the VOL table (volume, yesterday, trend, projection), an index with its Volume & cost tab, rolling volume and hourly profile, and the outliers model with its bounds.
Part 3, Tune, sets drops, spikes or both per index and in bulk, the inactivity policy for business hours versus nights, and the default KPIs and outliers scope. Part 4, Operate, covers the global license usage view (scopes, working days, days over quota, costs and chargeback), SPLK-VOL alerts with stateful emails, the AI Assistant and the AI ML Advisor review of a volume model.
Choosing the Volume tenant type¶
The tenant chooser opens from Virtual Tenants, Actions, Create a tracking tenant, and the Splunk Volume (Outliers) card sits second in it (callout 1). The wizard has five steps: basics, volume, inactivity, indexes & RBAC, review. The first step recalls the three arguments of the component (callout 2), minimal compute cost, outliers that scale safely, signal rather than noise, and what VOL provides: outliers on the licensed volume of every index, daily volume and month projection, local or remote, history backfilled on day one.
The tenant id (callout 3) is immutable, checked for availability as it is typed, twenty characters at most; volume-detection is used here. The alias and description (callout 4) appear on the Virtual Tenants card.
The Volume step and its defaults¶
The Volume step is the whole configuration and its defaults are the recommended values. Target environment (callout 1) is this Splunk or any remote account configured in TrackMe. Detect (callout 2) sets the default direction of every new index, drops and spikes, changed per index later. Backfill history (callout 3) is 30 days replayed at creation, capped by the retention of the license usage log.
The outliers scope (callout 4) covers every priority by default, because volume outliers are the point of this component. Minimum history (callout 5) is 15 days before a model reaches normal confidence; the notice (callout 7) confirms the backfill covers it. The default outliers KPI (callout 6) is the rolling 24h volume.
Settle margin and license search constraint¶
Expanding Advanced reveals two technical settings that normally stay at their defaults. The settle margin (callout 1), 300 s by default and adjustable from 60 to 3600, keeps the tracker behind the newest license events: events newer than now minus the margin wait for the next run, so late license events are still counted in their bucket.
The license search constraint (callout 2) is the base search every license search of the tracker is built from, indexed fields only. The default (callout 3) is index=_internal source=”*/license_usage.log”; it changes only to read a copy of the license usage events. Both exist for the rare deployment that forwards its license usage events elsewhere or delivers them late.
The inactivity policy by day and hour¶
Inactivity is the second detection: after how long without licensed volume an index turns red. The policy is variable by day and hour (callout 1), or static with one threshold. The timezone notice (callout 2) shows the server and browser clocks side by side: slots run in the server’s time and the editor converts browser hours on save. Quick templates (callout 3) cover working versus off-hours, weekdays versus week-ends and three tiers.
The default working_hours slot (callout 4), Monday to Friday 08:00 to 19:59 server time, uses 1d: a silent index is red the next day. Outside the slots (callout 5) the threshold is 7d. The policy applies live to every index without its own threshold.
Indexes, RBAC and creation¶
The last two steps are shared by every tenant type. Indexes & RBAC sets the TrackMe indexes (callout 1): summary, audit, metric and notable, where the defaults are fine. The Splunk owner (callout 2) is nobody by default; a service account is the better choice if the tenant will run AI automation. Roles (callout 3) default to trackme_admin, trackme_power and trackme_user, or any custom roles.
The review step is followed by Done, and the action progress events (callout 4) show the creation live as it happens: the tenant account, its reports and its trackers are created in seconds (callout 5), and the first collection runs straight away.
First collection and the pending backfill¶
The creation runs the first collection: the tracker seeds the last 24 hours of the license usage log and one entity per index appears, 43 enabled entities here (callout 1), each with its rolling volume. The backfill job plans itself at its first hourly run; until then the Virtual Tenants card reads Backfill pending (callout 2) and follows the backfill until it completes.
The Tenant Home header shows a matching vignette (callout 3): pending, then the percentage; a click opens the volume collection screen. Every index is green from the start (callout 4) because each had licensed volume in the last 24 hours. Volume 24h is already live (callout 5); the cost columns (callout 6) wait for the replayed history.
The volume collection screen¶
The volume collection screen explains both jobs, shown right after creation on the left and after the backfill on the right. Live collection (callout 1) runs every 5 minutes, incrementally, behind the settle margin; Run tracker now (callout 2) collects on demand. The backfill is Pending (callout 3) until the backfill job plans it at its next hourly run, then Complete at 100% (callout 4): the history is replayed day by day through the same code as the live collection, with historical timestamps.
The replay summary (callout 5) reports 30 days, 1,247 index-days and 358,104 metric points. The controls (callout 6) pause, resume and cancel; a restart extends the history backward. On completion every model is trained.
A late source and the collection¶
The Volume collection & history screen on a deployment whose license manager delivers its events late. Collected up to names the cursor (callout 1); the notice (callout 2) says that the license usage events are reaching the indexers late and that the collection waits for them instead of recording missing volume. Run tracker now collects on demand (callout 3) and About this deployment opens the license verification card (callout 4). The Historical license update (callout 5) is the post-upgrade replay that applied the licensed weight to the stored history, with its pause, resume, cancel and restart controls (callout 6).
Since 2.4.22 the window ends at the writer’s own watermark, each run rereads the last 30 minutes before its cursor, and every closed day is reconciled against license_usage.log after 03:00 at the license manager; a hold is discounted from inactivity, up to four hours.
Tenant Home after the backfill¶
Once the backfill has completed, the Tenant Home is the finished product. Two tabs (callout 1) hold the Volume Outliers indexes and the global license usage. The header counters (callout 2) show the entities and the green count, and the backfill vignette is gone. The charts by priority and by state (callout 3) show every index medium and green, and the key counts (callout 4) read 43 enabled and none in alert.
The entities overview (callout 5) is grouped by license pool, and the cost columns (callout 6), yesterday, average, trend and projection, are now filled from the replayed month.
Part 2 — Read¶
Part 2 reads what the tenant shows. The VOL table has one row per index with its volume, its trend and its projection. The entity screen of an index has the charts behind the figures: the daily licensed volume, the rolling volume windows and the hourly profile.
Behind each state sits the outliers model that turns a deviation into a state, with its bounds, its direction and its confidence.
Reading the VOL table columns¶
One row per index, grouped by the entity group with the group-by selector above the table (callout 1). The Detect toggle switches the direction in place (callout 2). Ingest (raw) 24h is the rolling raw ingest and Licensed yesterday the licensed volume of the last license day, both in binary units (callout 3); Avg daily and the 30-day trend give the baseline and its direction (callout 4), the month projection follows the weekday pattern (callout 5), Inactivity shows the threshold in force or the tenant policy (callout 6) and Outliers the model state (callout 7). Everything but the Detect toggle and the Inactivity column is computed by the collector every five minutes.
The Volume & cost tab¶
The entity opens on Volume & cost. The header gives the volume, the trend, the month, the inactivity and the status reason in words (callout 1). The key figures (callout 2) are the raw ingest of the last 24 hours, licensed yesterday, the 30-day average, the trend, month to date, the month projection and the pool used; the Detect control applies the direction at once (callout 3). The daily licensed volume chart (callout 4) draws one bar per license day over 60 days, in binary units, with the projection switch (callout 5) and the projected days as lighter bars (callout 6). About this deployment (callout 7) opens the verified days from the entity too.
Rolling volume and hourly profile charts¶
The rolling volume chart (callout 1) draws the last 7 days of the raw ingest, the KPIs a model can train on: 24 hours, 12 hours, 4 hours and 60 minutes, one unit per chart (callout 2). The outliers chart (callout 3) shows the model on the rolling 24-hour volume with its direction and anomalies, and the History slider widens the window; a value outside the learned band is a candidate outlier (callout 4). The hourly profile (callout 5) draws the daily shape of the index, which explains the band.
The outliers model of an index¶
The Outliers anomaly detection tab carries the model of the index on the rolling 24-hour volume (callout 1) over the time range picked, here the last 30 days (callout 2). AI ML Advisor and Manage outliers detection (callout 3) review it or tune it by hand. The counters give the impact score of an outlier (callout 4) and the outliers by bound, accepted, rejected and corrected (callout 5); the chart draws the KPI against its thresholds (callout 6). Because the history was backfilled, the model reached normal confidence on day one.
Part 3 — Tune¶
Part 3 tunes the detection. The defaults are sensible; these are the settings operators actually use: which direction matters for each index, when silence becomes an incident, which KPIs carry a model, and which indexes are in scope.
Every setting applies live, per index or in bulk.
Setting the detection direction¶
The Detect toggle in the table switches one index (callouts 1 and 2): both directions is the default, a metrics or development index is a spike concern, a security index a drop concern. The bulk edit category Volume Detection Direction (callout 3) applies Drops only, Spikes only or both to a selection (callout 4), with an update comment kept in the audit trail (callout 5). The direction is stored on the index’s outliers model and takes effect immediately.
Tenant inactivity policy and index overrides¶
Manage: Global inactivity threshold edits the tenant policy: variable per day and hour, or static (callout 1), with a time-zone notice that shows the server and browser clocks side by side because the slots are evaluated on the server (callout 2), quick templates (callout 3), the working_hours slot at one day (callout 4) and the threshold outside every slot, seven days (callout 5). The per-index modal names the slot in force and its source before anything changes (callout 6), offers the tenant policy, a static or a variable override (callout 7) and spells out what the index follows (callout 8).
Default KPIs and the outliers scope¶
Manage: Outliers default KPIs (callout 1) sets the rolling KPIs every new index gets a model on: rolling 24h, 12h, 4h and 60m, one model per KPI per index, the 24-hour volume by default. The shorter windows react faster, at the cost of more models. Also apply to the existing indexes (callout 2) adds the new models and removes the deselected ones, never touching the others.
Manage: ML Outliers tenant scope sets which indexes are trained and evaluated. Every priority is in scope by default (callout 3), and an optional filter expression (callout 4), such as object=siem-* NOT tags=lab, narrows on object, priority, tags, labels and pool. The other components’ ML filters do not apply to VOL.
Part 4 — Operate¶
Part 4 covers the global license usage view, alerting, and the AI Advisors. Global license usage (VOL) takes the whole environment’s licence from volume to cost.
It reads what the tenant already collected, with no monitoring, no alerting and no extra search on the license usage log, and answers the questions every Splunk administrator gets asked: who consumes, what a working day costs, which days went over quota, and who pays for what.
Global license usage overview¶
The Overview of the Global license usage view. Eight sub-tabs share one analysis period (callout 1): Overview, Daily changes, Investigate an index, Scopes, Working days and holidays, Over quota, Chargeback and Configuration, on rolling days, calendar months or the year to date. On this deployment the license manager delivers its events late, and the banner says that the collection waits for them rather than recording missing volume (callout 2). Licensed yesterday is the last closed license day in binary units, MiB, GiB and TiB by default, or Not recorded (callout 3); month to date and the projection sum the per-index figures (callout 4). With costs configured, the licence cost of the period and the cost per GiB appear (callout 5); with a quota in force, the quota used yesterday and the days over quota (callout 6). The insights start with yesterday against the previous day and the same weekday last week (callout 7).
The figures verified against the license log¶
About this deployment leads with the verification state. Daily figures successfully reconciled means that TrackMe’s totals and per-index figures match the detailed Usage records of license_usage.log (callout 1); the days where Splunk’s own RolloverSummary reports more or less than those records are listed with the difference in MiB and in percent (callout 2), then the range of days matched against the license manager (callout 3). The Deployment card names the platform, the Usage writer and the monitored indexes (callout 4); the Collection card the collected-through time, the license day frame and the unit labels (callout 5).
Technical details and verification dates lists the license source and one row per day (callout 6) with its status and revision (callout 7), both byte totals, the last successful reconciliation and the next attempt (callout 8).
Daily consumption and month-end projection¶
The first chart stacks the daily consumption per index (callout 3), each closed license day by consumer. The top consumers slider (callout 1) selects 5 to 100 series, with the rest grouped together, and the daily quota line (callout 2) shows the day the pool went over its quota at a glance.
The second chart shows the whole environment with the projection switch (callout 4): to the end of this month or of next month, the same rule as the entity chart. The projected days (callout 5) appear as lighter bars following the weekday pattern.
Consumers, costs, movers and pools¶
The top consumers table (callout 1) lists every index of the period; it sorts, filters and opens an index on click. With costs on, the cost columns (callout 2) give the cost of the period, of an average day and of a peak day, and the week-over-week and 30-day trend columns (callout 3) give each index’s direction.
The movers card (callout 4) turns the week-over-week comparison into rising, falling, new and stopped indexes: here siem-osnix-apac rose by 44 %. The license pool card (callout 5) gives the capacity view: meters for yesterday, the average and the peak day, the days over quota and the headroom.
Daily changes¶
Daily changes compares each closed license day with the previous day or with the same weekday a week earlier (callout 1). Filter indexes narrows the chart and the table by name, and the index selector focuses them on one index (callout 2). The cards give the environment volume on the selected day and its change (callout 3), then the largest increase and the largest decrease, each naming its index (callout 4). The chart ranks the top 3, 5 or 10 increases and decreases of each day, chosen from every index (callout 5); clicking a day opens the What changed on table with every index, sortable, with a seven-day sparkline (callout 6). Missing days stay gaps and a recorded zero is compared as zero.
Investigate an index¶
Investigate an index gathers what the tenant knows about one index, read-only. The selector lists every tracked index (callout 1); a row of Daily changes, a top consumer or a deep link arrives here too. The index’s pool and scopes (callout 2) and the Incidents & entity settings, Search licensed volume and Search index events buttons (callout 3) sit above the licensed volume and peak day of the analysis period, with the cost when costs are on (callout 4). The selected day is compared with its comparison day (callout 5), followed by the daily changes of this index alone, its daily volume with the projection, the rolling volumes, the outliers model and the hourly profile (callout 6).
Named scopes¶
A scope is a set of indexes analysed together. The second row holds All scopes and one tab per enabled scope, a filterable selector once they no longer fit (callout 1). The scope’s source is index patterns or the CIM datamodel macros, read on this search head or through a remote account on the Enterprise Security tier (callout 2). Its tab gives the licensed volume, the share of the licence or the cost of the period (callout 3), the daily average with the working-day average (callout 4), the peak day, the last 7 days against the 7 before, the month projection, and the daily volume stacked by index with the scope’s own projection below (callout 5).
All scopes side by side¶
All scopes compares every enabled scope side by side (callout 1): its source, its indexes, the licensed volume of the period, the share of the licence or the cost (callout 2), the last 7 days against the 7 before and the month projection. The headers sort and the filter narrows by name; a row opens the scope. An index may belong to several scopes, so the shares can add up to more than 100 % (callout 3); the share bars are scaled to the largest scope (callout 4). Configure opens the scopes configuration (callout 5). The view is computed from the per-index search the Overview already runs, so it carries no daily average or peak.
Working days and holidays¶
The Working days & holidays sub-tab separates working days, weekends and holidays. The tenant’s calendars and working weekdays are named (callout 1); the working-day average leaves weekends and holidays out (callout 2) and the view says how much more a working day consumes than a non-working one (callout 3); the holidays of the period are listed with their calendar (callout 4). The daily chart colours each license day by kind of day (callout 5) and the average per weekday leaves the holidays out (callout 6).
Days over quota¶
Over quota needs a quota in force: the license manager’s pool size on Splunk Enterprise, or a daily quota configured per pool. The window covers 3, 6 or 12 calendar months (callout 1); the cards count the days over quota in the window, this month and last month, per pool (callout 2), the rolling 60 days Splunk counts warnings on (callout 3) and the worst day as a percentage of the quota (callout 4). The chart draws each day against the quota, red over it and blue within (callout 5). These are days over quota, not Splunk license warnings.
What made the day¶
What made the day compares the pool’s largest indexes on the day picked with their usual day, their average over the license days they recorded before. The day selector defaults to the latest day over quota (callout 1), Previous day and Next day step through the window (callout 2), and the badge says by how much the day was over or within its quota (callout 3). The chart draws that day against the usual day per index (callout 4), the table reads the same comparison in figures with the share of the day (callout 5), and the list of days over quota opens any of them (callout 6).
Chargeback statement¶
The Chargeback sub-tab turns the licence into a monthly statement per cost centre. The statement month covers thirteen calendar months (callout 1) and Export CSV keeps a closed month as charged (callout 2). The cards give the bill of the month (callout 3), the volume charged to the centres and the unallocated volume kept apart (callout 4) and the centres over their monthly budget (callout 5). The statement lists each centre’s volume, share, cost, budget used and top indexes (callout 6), followed by the cost per cost centre month by month. An index belongs to the first centre that matches it, so every GiB is charged once.
Configuring the license view¶
The Configuration sub-tab holds six inner tabs (callout 1): scopes, working days, daily quota, chargeback, costs and Import / Export, saved together with a revision so two people editing at once never overwrite each other. The chargeback section is enabled with a switch (callout 2), keeps the unallocated volume as its own line or spreads it (callout 3) and holds the ordered cost centres with an owner, a monthly budget and include and exclude patterns, previewing the indexes each resolves to (callout 4). The costs section holds an annual licence cost with the licensed volume per day or a cost per GiB (callout 5), the allocation by rate or by share of the bill (callout 6) and the price history, so a renewal never reprices past months (callout 7).
Moving the configuration¶
Import / Export, an inner tab of the configuration (callout 1), moves the whole document between tenants and deployments. Export downloads the saved configuration as a JSON file: scopes, working days, quotas, costs with the price history and chargeback, never unsaved edits (callout 2). Import reads such a file, validates it on the server, shows the summary and the JSON, and Replace configuration replaces the destination’s document after a confirmation that names the tenant, with no merge (callout 3). Remote CIM accounts and Bank Holidays calendars are referenced by name and must exist on the destination.
The AI Assistant on the license view¶
While the license view is open, the Tenant Home AI Assistant works on it. Asked what made the pool go over its quota on 14 September, it answers with the figures of the sub-tab on screen (callout 1) and names the culprit (callout 2), siem-infra-emea, from the day selected in What made the day (callout 3).
Asked to add a cost centre for the cribl_* indexes, it proposes the change through the Concierge (callout 4): one configuration patch with its endpoint and permission (callout 5), applied only on Confirm and run (callout 6), audited, and reloaded by the view.
Configuring alerting for red indexes¶
Volume Outliers alerts like any TrackMe component. The SPLK-VOL (Volume Outliers) alert type (callout 1) generates the notable events and acknowledgments for red indexes; its defaults (callout 2) are any priority, red, all indexes, outliers on. The outliers trigger stays on because the outliers are the state of this component; the same alert covers inactivity.
A TrackMe stateful alert, volume-alert here, sends the opened, updated and closed emails. Emails and Ingest (callout 3) delivers emails plus stateful events; the delivery account (callout 4) is any email account in TrackMe; the environment name (callout 5) is stamped in the email header; the recipients (callout 6) sit further down with the charts and the optional AI status report.
Launching the AI ML Advisor¶
From the Outliers tab of an index, the AI ML Advisor runs as a background agent job with read tools: the entity, its model rules, the outlier and score history, the training details. The AI provider (callout 1) is any provider configured in TrackMe. Inspect or act (callout 2) chooses between a review that changes nothing and a remediation applied after consent.
Additional instructions (callout 3) carry the operator’s intent in words: this security index is monitored for drops only; does the model fit the weekly business-hours pattern, and would the week-end lows raise false drops? Start Analysis (callout 4) launches the job, and the live progress feed (callout 5) streams each tool call as it runs.
Reading the advisor verdict¶
The verdict is Healthy (callout 1): green, with no outlier and no false positive in the last 30 days. The summary (callout 2) answers with evidence from the model itself: it uses weekday × hour seasonality, 168 groups, all fitted, so the week-end lows are expected rather than drops, and its bounds track the weekday peaks and the week-end troughs. The two recommendations (callout 3) are low: no action needed, the drops-only direction confirmed.
The advisor explains why nothing changes rather than rationalising a recovered metric; the reasoning trace shows every step. Act mode would apply period exclusions, retraining or bound changes after consent. The automated batch runs the same review nightly over the in-scope indexes.
Running cost of the component¶
Collection is one bounded search per tenant every 5 minutes, reading only the new license usage events, whatever the number of indexes: no per-index and no per-sourcetype scan. Metrics cost about 216 KB per index per day of metric-index licensing; the backfill writes the same series over the chosen depth, and the wizard shows the estimate.
Models are one per index on the rolling 24 hours; the VOL scope and the default KPIs control how many. Live and backfill never overlap, the window waits for a late source and every closed day is reconciled against the license usage log, and a failed search never advances the cursor, so missing volume is never recorded as a drop. The rolling values come from the buckets TrackMe holds, not from its own metrics.
Where we are¶
One tenant watches the licensed volume of every index. The backfill makes it useful on day one; the table, the entity screen and the model explain every figure, in binary units and verified against the license usage log every day; the direction per index, the inactivity policy with its overrides, the default KPIs and the scope make the detection the operator’s own. The Global license usage view, from the verified figures and the daily changes to the scopes, the working days, the days over quota, the costs and the chargeback, SPLK-VOL alerting with stateful emails and the AI Advisors take the component into operations.