Volume Outliers — in depth

Tip

This is the in-depth reference for the Volume Outliers (splk-vol) component. For the readable overview of what VOL is and when to use it, start with VOL — Volume Outliers; for a guided, end-to-end walkthrough on a live deployment, read the Volume Outliers white paper. This page covers the collector, license days and time zones, the closed-day reconciliation, the backfill, the outliers and inactivity policies, the projection, the Global license usage view and its configuration (scopes, working days, daily quotas, costs, chargeback), alerting, the REST endpoints, the metrics and troubleshooting.

The entity model

• An entity is one Splunk index: object is the index name, object_category is splk-vol, and the entity’s group is its license pool (a single-pool deployment has one group; on Splunk Cloud, where a pool has no limit, the group is default). An index reported under several pools follows its primary pool — the one carrying most of its bytes in the window — with hysteresis: it keeps that pool up to 24 hours while the pool reports it, and moves only when another pool carries twice its volume.

• The state is green by default: collection never turns an index orange or red on its own. The state comes from ML outliers (a lower-bound breach is a drop, an upper-bound breach a spike) and from inactivity, through the same hybrid impact scoring as every component (Entity State & Scoring). The tenant impact keys are impact_score_vol_outliers and impact_score_vol_inactive (default 100: an inactive index turns red).

• Per index the analyst chooses the detection direction — Detect drops, Detect spikes, or both — with the inline Detect toggle of the VOL table, in bulk edit, or in the entity’s Volume & cost tab. The direction is stored on the index’s outliers model (alert_lower_breached / alert_upper_breached); the tenant default is vol_default_direction (both).

• Everything generic applies: priority, tags, labels, SLA, logical groups, disruption queue, acknowledgments, notes, maintenance, monitoring time, enable / disable, deletion. An index deleted temporarily keeps its bucket history when it had licensed volume in its last 24 hours (it comes back with its true rolling volume); a permanent deletion drops it and the index is never collected again.

The collector

One report per tenant, trackme_vol_tracker_tenant_<tenant_id>, runs every 5 minutes through the standard tracker executor (execution summary, licensing checks and the scheduled-time anchor come with it). Each run:

1. Loads the checkpoint of the tenant — the cursor (the end of the last finalized 5-minute bucket), the validated history start, the license manager’s UTC offset and the deployment type.

2. Establishes the license writer — the one host writing type=Usage over the last 3 days — and scopes every license search to it. No writer, or several (two license managers in one license search, a test stack forwarding its log to the same index), holds the collection with the VOL_LICENSE_WRITER_UNKNOWN / VOL_MULTIPLE_LICENSE_WRITERS diagnostic rather than summing licences: the cursor does not move, the history is kept, and the license view and the Manage: volume collection screen say so. Scope vol_license_search with host=<license manager> to resolve it; after a license manager is replaced, collection resumes by itself once the previous host has written no Usage for 3 days.

3. Computes the window [cursor, floor(min(now, watermark) − settle)), aligned on 5-minute boundaries. The source watermark (2.4.22) is how far the writer has delivered: the newest _internal event of the writer host, which travels the same forwarding path as the license usage log (for a custom license search, the earlier of that and the newest license event). The window waits for a late source instead of closing on events that have not arrived; the settle margin (vol_settle_seconds, default 300 s) still leaves the time to write every Usage event of a bucket before it is read. A run catches up at most 6 hours at a time; a gap above 24 hours re-seeds the history (see Validated history below).

4. Runs one bounded search on the license usage log, local or on the tenant’s remote account, starting vol_late_overlap_seconds (default 1800 s) before the cursor so stragglers of the previous buckets are read again:

   search (index=_internal source="*/license_usage.log" host=<writer>) type=Usage idx=* earliest=<cursor − overlap> latest=<end>
   | bin _time span=300s
   | eval effective_bytes=b*coalesce(licensed_weight,1)
   | stats sum(b) as b, sum(effective_bytes) as effective_b, latest(poolsz) as poolsz by _time, idx, pool, host
   

   plus, once a day (and on every run around the license manager’s midnight), a cheap indexed probe on the RolloverSummary event that returns the license manager’s UTC offset and the deployment type. b is the raw ingest; effective_b the licensed volume — data Splunk licenses at a reduced weight carries a licensed_weight on its Usage rows (2.4.22). The 5-minute buckets and the rolling values the outliers models train on stay raw ingest; the daily figures (yesterday, month to date, trend, projection and the whole Global license usage view) are licensed volume, and the raw ingest is shown next to Licensed yesterday when the two differ.

5. Updates the ring of each index — the last 288 buckets (24 hours) kept in the tenant’s KV store: appends the new buckets (a bucket with no event is 0), recomputes the rolling sums at each bucket end, accumulates the current license day and closes any license day that ended. A bucket read again through the overlap is raised to the re-read total, never lowered (license events are never removed, so a re-read cannot take volume away); the difference goes to the ring and to the open or closed day it belongs to, and the 5-minute and rolling points already written are not written twice.

6. Writes the metrics with explicit bucket timestamps, then advances the cursor — the cursor moves only once the metrics and the rings are persisted, so a run that dies mid-way is replayed, never skipped.

7. Emits one entity row per known index, carrying the latest rolling values, the daily figures (yesterday, month to date, 30-day average, trend, projection), the pool usage on Splunk Enterprise, the detection direction, the effective inactivity policy and the hold the run measured on the source (see Late license events below).

8. Reconciles closed days — after 03:00 license-manager time, up to two of the last seven closed license days are reread from the license usage log, repaired if needed and compared with the license manager’s own daily summary (see Closed-day reconciliation).

Note

Why a ring instead of querying the metrics index. The rolling volume must not depend on the metrics pipeline’s own indexing latency: a blocked queue would read as a drop and raise false lower-bound outliers. The rolling values are computed from the buckets TrackMe holds and then written as metrics.

Integrity rules

• Validated history. Every ring and the tenant checkpoint carry the time from which the history is continuous. A rolling KPI is emitted only once its whole window lies inside the validated history, and a license day is reported only when it was observed from its start — the first run’s 24-hour seed writes the 5-minute volumes for every bucket, but the 24-hour trend only from the last seed bucket. No ramp-up or gap ever reaches the series the models train on.

• Search errors fail the run. An error from the usage search — or a warning saying the results are incomplete (a peer down, a search process ended prematurely) — stops the run before anything is written; the cursor does not advance and the missing volume is never recorded as a drop.

• One collecting run at a time. A lock protects the collection; an overlapping run (for instance run the tracker now during the schedule) rebuilds the entity rows from the stored rings without writing any metric point. A run that meets the lock of a tenant being seeded waits for it (up to 4 minutes) and then collects normally.

• A new source re-seeds. Changing the target environment (vol_account) or the license search constraint discards the rings and the checkpoint and re-seeds 24 hours from the new source, emitting only the points after the previous source’s cursor — a series never holds two points for the same bucket. The previous source’s indexes stop being updated, turn inactive and can be deleted.

• Fail-closed reads. The rings, the per-index directions and the permanently deleted indexes are read without a fallback: a failed read fails the run instead of resetting a direction or resurrecting a deleted index.

• Squashed rows keep the index. When the license manager squashes Usage rows (squash_threshold), it drops the host and source, never idx — squashing never reads as a drop.

• One license writer. Every search is scoped to the host established as the Usage writer; a held collection (no writer, several writers) never sums volumes and never moves the cursor.

Late license events

The license usage log reaches the indexers through the license manager’s own forwarding. A blocked output queue on the license manager, an indexer restart or a license manager clock behind the indexers used to turn into a sudden, lasting drop on every index at once while the events were all there, just late. Since 2.4.22 three layers keep the figures exact whatever the latency:

• The window waits for the source — it ends at the writer’s watermark (step 3 above), so on an in-order forwarding path every bucket the collector reads is complete.

• The late overlap — each run rereads the last vol_late_overlap_seconds before its cursor (default 30 minutes, 0 disables, up to 4 hours) and raises the buckets it finds short, never lowers one.

• The closed-day reconciliation — a straggler that arrives after its license day closed is repaired in the day’s figures by the reconciliation (next section).

A window held 15 minutes or more behind the wall clock raises the VOL_LICENSE_SOURCE_DELAYED diagnostic in the license view: the collection is waiting for the events instead of recording missing volume — check the license manager’s forwarding to the indexers and its clock. The hold each run measures is written on every index and discounted from the inactivity clock, up to 4 hours, so a late source does not turn every index red; a source delivering nothing for more than 4 hours is an outage and does. The measure is the run’s, not “now minus the cursor”: a tracker that stops running keeps its last measure and the inactivity clock keeps running.

License days and time zones

Splunk measures the license on license days that roll at the license manager’s local midnight, whatever the time zone of your search heads or of your Splunk user. VOL follows the license manager:

• The daily probe reads the license manager’s UTC offset from the RolloverSummary event (its raw timestamp carries the offset, and its time is the true day boundary). On a deployment younger than a license day no rollover has been logged yet: the offset is then read from the raw timestamp of the latest Usage event (every line of the license usage log carries it) and the Manage: volume collection screen says so. UTC is assumed only when there is no license event at all, and the open day is voided when the real offset turns out to differ.

• The probe runs on every run between 22:00 and 03:00 license-manager time, so a daylight-saving transition is caught within the run that follows it. The buckets between the old and the new day boundary move to the day they belong to, and a corrected closed day is re-emitted (a fall-back day lasts 25 hours, a spring-forward day 23 hours).

• The daily_volume_mb metric of a license day is stamped in the middle of that day, so a chart binned by day places it on the right date whatever the viewer’s time zone.

The daily figures (yesterday, month to date, the trend and the projection) refer to license days, and the Volume & cost tab says which license date “today” is for the collector.

Closed-day reconciliation

The live collection reads each 5-minute bucket once, shortly after it ends. A Usage event that reaches the indexers later than the settle margin (a blocked forwarding queue on the license manager, an indexer restart) would be missing from the day’s figure, and Splunk’s own daily summary — the RolloverSummary event the license manager writes at its midnight — can itself differ from the sum of the detailed Usage records. Since 2.4.22 the tracker therefore reconciles every closed license day against the license usage log and says what it found.

How it runs

• The reconciliation is a phase of the 5-minute live tracker, under the collection lock: a run that collects (one Usage writer identified, no diagnostic hold) checks at most two closed days among the seven most recent the tenant holds, in due-time order, so a day that keeps failing never starves the others.

• A closed day becomes due at 03:00 in the license manager’s time zone of the following day, leaving late events and the rollover itself the time to be indexed. A verified day is checked again every day at 03:00 while it stays in the seven-day window (a late redistribution between indexes is caught); a day that could not be read, or whose totals disagree, is retried every hour.

• Each attempt reads the day’s RolloverSummary (the day’s exact bounds come from the two adjacent rollovers and their UTC offsets, so a 23-hour or 25-hour day is read as such), then rereads the whole day per index from the Usage records — the same source and the same remote account as the live collection, under the tenant’s license search constraint.

• A reread that answers repairs the stored figures of the day: the raw ingest and the licensed volume of every index of that day, including an index that stopped consuming (its day is zeroed, never left at a stale value) — never an index first seen on a later day, never a permanently deleted index. The repair happens whether or not the rollover agrees: the detailed Usage records are the measurement.

• A repaired day is re-emitted: one corrected daily point per index (daily_volume_mb, daily_ingest_mb), stamped later than the original point inside the same license day and carrying the license_day and reconciliation_revision dimensions (r1, r2…). The entity searches read the daily series with latest(), so a corrected point replaces the original one in every chart and figure without summing the two; the stored daily figures the license view reads carry the same snapshot.

• Nothing is invented: when the rollover or the Usage records of the day are no longer available (retention), the day boundaries cannot be established, or the search fails, the previous figures are kept untouched and the attempt is retried an hour later. A day that leaves the seven-day window without a successful reread keeps its figures and is marked as outside the automatic window; it is never set to zero.

Verified, or reconciled with a difference

Repairing the figures and agreeing with the license manager are two different things. Once the day is reread, its raw and licensed totals are compared with the RolloverSummary totals to the byte (8 bytes of tolerance, independent of the displayed rounding):

• Verified — the Usage records and the rollover agree (Verified for collected scope when permanently deleted indexes are excluded from the comparison).

• Source reconciled · summary differs — TrackMe matches the detailed Usage records, but Splunk’s own RolloverSummary reports more or less than those records. The card gives both totals, the difference in MiB and in percent of the summary, and the next comparison check. This is information, not an alert: TrackMe’s reconciliation is complete, the figure is the one the Usage records measure, and the comparison keeps running while the day stays in the window.

• Reconciliation incomplete — the day could not be reread (the reason and the last attempt are on the card) and keeps its previous figures; the retry is automatic.

Where to see it

About this deployment — the button at the top of the Global license usage tab, and the same card on the Manage: volume collection screen — leads with the verification state: N closed license days verified, Daily figures successfully reconciled with one line per day whose summary differs, or Collection or verification needs attention with the days to look at. Technical details and verification dates lists every day of the window with its status, revision, last check, both byte totals and the next check. The cards of the Global license usage views whose figure comes from the license usage log offer Verify against license_usage.log in their menu: the search that sums the Usage records of that exact day and scope, so you can confirm the figure yourself.

Note

After an upgrade to 2.4.22. Nothing to request and nothing to wait for: the closed days are those the tenant already holds, so the first collecting runs after the upgrade reconcile the last seven of them, two per run. There is no manual reconciliation request; Run tracker now on the Manage: volume collection screen runs the same tracker at once, under the same rules — it cannot pick a day, nor reconcile a day before 03:00 license-manager time the day after it closed.

Backfill

At creation the tenant requests a backfill of its history (vol_backfill_days, default 30) so the outliers models train as soon as it completes and the daily history, the trend and the month projection are complete. The wizard offers No backfill, 7, 14, 30, 60 or 90 days, or Custom… for any whole number of days up to 400 — 13 calendar months, the depth of the chargeback statements. It caps the depth to the retention of the license usage log it can reach, shows the reachable span, and shows the estimated cost in metric points before you confirm.

• The backfill is a separate report, trackme_vol_backfill_tracker_tenant_<tenant_id>, scheduled hourly while a plan is open and un-scheduled once complete. It walks forward in time (every rolling value needs the 24 hours before it) in day-sized chunks, each chunk replaying exactly the live collection with historical timestamps, and runs for at most 50 minutes per slot.

• The backfilled range ends where the live collection started: the live tracker and the backfill never overlap and no point is written twice. The first backfilled day is aligned on a license-day start, so it is complete; every complete license day of the range emits its daily_volume_mb.

• The live collection emits a rolling value only once its whole window lies in its own history, so the first live day has no 24-hour value (and no 12-hour, 4-hour or 60-minute one for its first hours). About 24 hours after the live start, the hourly backfill job replays that day and writes the missing rolling values; until then the Manage: volume collection screen reads First live day trends: pending and the rolling chart shows a gap at the seam.

• Daylight-saving transitions inside the range are replayed with the offset in effect at the time, from the RolloverSummary history.

• On completion the backfilled days are merged into each index’s live history (the entity figures reflect them at once), the seam day around the live start is completed, and a training of every VOL model is requested, so outliers are usable without waiting for the next training cycle.

Progress is visible in three places — the tenant card on the Virtual Tenants page (Backfilling 42 %), the vignette in the Tenant Home header band, and the Manage: volume collection screen (live cursor, validated history, license-manager offset, deployment type, the backfill progress with pause / resume / cancel / restart, and run the tracker now). From the creation until the backfill job’s first hourly run, the three read Backfill pending with the requested depth and the job’s next run.

An upgrade to 2.4.22 runs a Historical license update through the same job: the reachable source history is reread once to apply the licensed weight to the stored daily volumes (Updating history on the vignette, Historical license update on the collection screen, with the same pause, resume and cancel). The raw rolling history and the outlier models are untouched; a day whose source has expired keeps its raw value, and the Adjusted history from date on the screen says where the licensed history starts.

Note

A restart re-plans only the history before the earliest point already written: it extends the history backward and never rewrites it. A change of the target environment or of the license search constraint while a plan is open cancels the plan (the new source is re-seeded first); restart it once the live collection runs on the new source.

The license search constraint (advanced)

Every license search of the tracker, the backfill and the probes is formed from one tenant-level constraint, vol_license_search. Its default targets the license usage log of the deployment the tenant collects from:

index=_internal source="*/license_usage.log"

You normally never change it. The Advanced section of the wizard’s Volume step (and the Configure tenant form afterwards) lets you point the tenant at another data set holding events in the license_usage.log format — for instance a copy of the license usage events forwarded to a dedicated index, or a deployment where _internal is not reachable under that name. The constraint is a plain base search on indexed fields:

• no pipe, subsearch (square brackets), macro (backtick), line break, earliest or latest — the tracker sets the time range;

• it must name an index as a real search term (index= inside a quoted value does not count), and not every index: a bare wildcard (index=*) is refused, index=lic* is fine;

• every OR alternative must be index-scoped (index=a OR index=b is fine, index=a OR source=x is refused) and parentheses must balance.

The wizard validates the constraint as you type, and it is validated again at every save.

Important

The data behind a custom constraint must have the license_usage.log key=value fields extracted at search time — type, idx, b, pool, poolsz — since the tracker filters on type=Usage idx=* and sums b. The real license_usage.log in _internal (sourcetype=splunkd) is extracted out of the box; a copy stored under a sourcetype with KV_MODE = none (stash, for instance) needs a props.conf stanza with KV_MODE = auto for its source, or the tracker reads no usage row at all.

Changing the constraint on a running tenant re-seeds the collection from the new source (see Integrity rules above) and cancels an open backfill plan.

Outliers

VOL rides TrackMe’s native outliers engine (Machine Learning, Outlier detection — in depth) with component-specific defaults:

• Every new index gets one model per default KPI — vol_outliers_kpis, default last_24h_volume_mb, any subset of the four rolling KPIs. The rolling 24-hour volume follows the daily pattern and is the proven choice; the shorter windows (12 h, 4 h, 60 min) react faster to a sudden drop or spike and are noisier. One model per index is the deliberate default: several models per index multiply the noise. The models use the day-of-week × hour-of-day time factor over a 90-day calculation period.

• The default KPIs are set in the wizard and changed later in Tenant Home (Manage: Outliers default KPIs), with an opt-in apply to the existing indexes that adds the models of newly selected KPIs and removes the models of deselected rolling KPIs — models of kept KPIs and of non-rolling metrics are untouched. A model can also be added on, or switched to, any of the four KPIs from the entity’s Outliers tab.

• Direction — a drop is a lower-bound breach, a spike an upper-bound breach; the per-index detection direction enables one or both bounds on the model.

• Scope — VOL has its own scope: vol_mloutliers_priority_filter (default: every priority — volume outliers are the point of the component) and vol_mloutliers_filter_expression (the TrackMe filter DSL on object, priority, tags, labels and pool). The tenant-wide mloutliers_priority_filter and mloutliers_filter_expression do not apply to VOL entities. The Manage: ML Outliers scope modal edits the VOL pair on a VOL tenant, and the table’s Outliers column shows out of scope with the reason for an excluded index.

• Minimum history — the wizard exposes the tenant override of splk_outliers_min_days_history (15 days on a VOL tenant by default, against a system default of 30) and tells you whether the requested backfill covers it: with 30 days of backfill, the models reach normal confidence once the replay completes.

• Reading a model — the entity’s Volume & cost tab draws one outliers chart per enabled model, each with its own History slider, from 24 hours (the default) to 90 days. The chart reloads once, when you release or leave the slider, and its subtitle gives the number of anomaly windows over that range with their points (for instance 5 anomaly windows (36 points): consecutive anomalous points form one window, as the chart’s legend counts them) and the model’s confidence.

• ML is always on for VOL. A tenant where ML is off for the other components because it only runs DHM / MHM keeps ML on for VOL alone (mloutliers_allowlist=vol).

• The AI ML Advisor — interactive and its automated batch — covers VOL models like any other (AI ML Advisor).

Note

A one-off spike right after a short backfill. The models use the day-of-week × hour time factor over a 90-day period. Trained on a short history, a model learns a single spike as a weekly pattern: its bounds widen on that weekday and hour until the day ages out of the calculation period. To remove it sooner, add a period exclusion on the entity’s Outliers tab and retrain, or let the AI ML Advisor in act mode do it.

Inactivity policy

An index whose last licensed volume is older than the threshold in force turns red (anomaly_reason=inactive, impact_score_vol_inactive). Two policy shapes exist:

• static — one threshold (vol_max_sec_inactive, default 7 days);

• variable — slots by day of week and hour of day, the DSM / DHM variable-delay model: {slot_name, days (0 = Monday .. 6), hours (0-23), max_sec_inactive}; the first matching slot applies, the default (vol_inactive_default) outside every slot.

The tenant policy (vol_inactive_policy, variable by default: 1 day during working hours, Monday to Friday 08:00–19:59, and 7 days for nights and week-ends) applies live to every index without its own override. A per-index override — static or variable — is set from the entity menu (Modify → Inactivity threshold), from the bulk edit category Inactivity Threshold, or by clicking the Inactivity column, and can be sent back to the tenant policy. A threshold of 0 disables the check (the whole static policy, or one slot); otherwise it runs from 300 seconds to 90 days. Durations accept seconds, a unit suffix (30m, 12h, 1d, 1w) or [D+]HH:MM:SS.

Note

The slot clock is splunkd’s system time zone — the frame the editor’s hour grid is drawn in (the editor shows the server time and converts your browser hours). It is not the time zone of the user running the tracker or viewing the page, so the scheduled tracker, a Tenant Home refresh and an inactivity save all pick the same slot.

The red status message names the slot in force (and says set for this index when an override applies); the table’s compact Inactivity column shows the threshold in force, with the slot and the source in its tooltip.

Trend and projection

The collector computes, per index and on every run, from the closed license days it holds:

• yesterday_volume_mb, mtd_volume_mb (month to date), avg_daily_30d_mb;

• trend_30d_pct — a robust (Theil–Sen) slope over the last 30 closed days (the median of the slopes between every pair of days), as a percentage of their median, never below −100 % (a volume cannot fall by more than all of it), from 7 days of history;

• projected_month_mb — the month to date plus, for each remaining day, its expected rate: the median of the same weekday over the last 28 license days once two samples exist, else the mean of the last 7 days. The open day counts for the larger of its volume so far and its expected rate. No projection before 7 closed license days: a day or two of history gives no sensible monthly estimate. A day missing inside the index’s history (a collector outage past the catch-up) is estimated at its rate; the days before the index’s first recorded day are not.

Both figures use medians on purpose: a single outlier day cannot dominate them. A one-off spike or outage inside the 28 days is not repeated on every remaining same weekday of the month, and does not turn a flat index into a steep trend. With only two samples of a weekday, the median is their mean.

The Volume & cost tab draws the daily licensed volume of the last 60 days with the per-day projection to the end of this month or of next month (a switch remembered per viewer), scaled to the collector’s figure so the chart and the card agree; Open in search carries the same rule in SPL. There is deliberately no predict: a weekday-weighted rate is what the license bill follows.

Global license usage (VOL)

A Tenant Home tab next to the VOL component tab: the license consumption of the whole environment, with an opt-in configuration layer (named scopes, working days, daily quotas, costs and chargeback). It reads what the tenant already keeps (one daily point per index and closed license day, and the entity figures), so there is no monitoring, no alerting and no search on the license usage log: a few mstats searches, the same for a remote deployment. It is as deep as the tenant’s history, live collection plus backfill.

Principles

• Closed license days only, in the license manager’s time zone. Today, the open day, is never in a chart; the month figures (month to date, month projection) come from the entities.

• One value per index and license day, placed in the middle of the day, so any viewer within 12 hours of the license manager sees it on its own date.

• Analysis period: every view has its selector — rolling 7, 14, 30, 60 or 90 closed license days, 6 or 12 calendar months, the current or previous week, or the year to date (Overview opens on 30 days, Daily changes on 14, Investigate an index on 90). A banner says when a backfill is still filling the history in, or where the history starts when it is shorter than the period.

• Binary units, IEC labels. Every volume is computed in base 2 (1024) and labelled MiB / GiB / TiB by default (2.4.22). The system setting [trackme_ui_defaults] volume_unit_labels = splunk (System settings → User interfaces) switches the whole app to Splunk’s own MB / GB / TB labels for the same values, and each user can pick either style, or the system default, in the settings panel of Tenant Home and Virtual Tenants; the values never change, only the labels. Every chart draws in one unit picked from its largest value; tables and tiles follow the same rule.

• Exact dates, stated frame. Licensed yesterday and the pool meters show the last closed license day or Not recorded (the last recorded day is named separately), never an older day under that label; every sub-tab states the license-day frame (license manager midnight and its UTC offset) and names the collection scope — a custom license source, permanently deleted indexes, missing or partial days.

• No auto-refresh. Tenant Home’s auto-refresh is paused while the license view is open, so an analysis or a configuration draft is never reloaded under you; Refresh reloads.

Sub-tabs and deep links

Two rows of sub-tabs: the views on the first — Overview, Daily changes, Investigate an index, Scopes, Working days & holidays, Over quota, Chargeback, Configuration — and, under Scopes only, a second row with All scopes and one tab per enabled scope (tabs while they fit, a filterable selector beyond, up to 50 scopes). Each view has a deep link:

TenantHome?tenant_id=<tenant>&component=vol&tab=license_usage&view=<view>

where <view> is overview (the default), daily-changes, investigate-index, scopes, scope-<scope id>, working-days, over-quota, chargeback or configuration. A link to a scope that no longer exists, or is disabled, opens the overview. Daily changes and Investigate an index keep their selection in the link — vol_index=<index>, vol_day=YYYY-MM-DD and vol_compare=weekday (the previous-day comparison is implicit) — so a dated drilldown can be shared; without a date, a link opens on the last closed day against the previous day.

Overview

• Key figures — Licensed yesterday, Daily average (with the working-day average once working days are classified), Peak day, Trend (the 30-day rule of the entities, over the environment), Month to date, Month projection (the sum of the per-index projections, with the number of indexes that have one), Top 5 share. With costs on: the Licence cost or the Cost of the volume of the period (see How costs are computed), the Cost per GB, and the Unused capacity (or Over the licence). With a quota in force: the Quota used yesterday and the Days over quota of the period.

• Insights in plain words, most important first: a Yesterday block (the licensed volume of the last closed day, its change against the previous day and against the same weekday last week, the top changes per direction), then days over quota and pools close to it, the trend, the largest increase and decrease, indexes that stopped consuming, new consumers, the concentration, the weekly pattern and the licensing model.

• Every card’s menu offers Refresh, Open in search (the mstats search behind the figure) and, for a figure measured in the license usage log, Verify against license_usage.log (see Closed-day reconciliation).

• Licensed volume per day — stacked by top consumers, with a Top consumers slider from 5 to 100 (remembered per viewer; the rest is All others) and the daily quota as an overlay line when a quota is in force.

• The whole environment per day — the daily total with the projection to the end of this month or of next month, the same rule as the entity chart.

• Top consumers — every index, sortable and filterable: rank, share, total, average, peak, the last 7 closed days against the 7 before, pool, projection and 30-day trend, plus the cost columns when costs are on. A click opens the entity.

• Movers — rising, falling, new consumers and stopped consuming, last 7 closed days against the 7 before. They need 14 recorded days: on a young tenant the card says so instead of calling every index new.

• License pools — one card per pool with its meters (yesterday, average, peak day), the number of recorded days over quota and the headroom; (configured) marks a quota set in the configuration. Each pool’s quota is read from the pool’s own rows, and the view lists the pools configured now: a pool the license manager has not reported for 4 days leaves the quotas (the card says so after a day without report — a pool fed on weekdays only reads so every weekend) while its recorded days stay in the charts under its name.

Daily changes

Which indexes moved, day by day. Each closed license day of the analysis period is compared with the previous day (the default) or with the same weekday last week:

• Cards — the environment volume on the selected day and its change against the comparison day, then the largest increase and the largest decrease of that day, each naming its index.

• Largest daily volume changes by index — one bar per mover and day, above zero for an increase and below for a decrease: the top 3, 5 or 10 per direction, chosen each day from every index (never only the Overview’s top consumers). Hover a bar for its index and percentage; click a day to select it.

• What changed on <day> — every index of the selected day: comparison volume, selected-day volume, change, change in percent and the recorded volume of the last 7 days as a sparkline; sortable (the Change header cycles largest change → increases first → decreases first) and paged by 25. A row opens Investigate an index on that index and day.

• Filter indexes narrows the chart and the table by name, and the index selector next to it focuses the cards and the chart on one index, without the ranking cap.

Missing days stay gaps and a recorded zero is compared as zero: a change needs a recorded volume on both days, and an index with none on the comparison day reads No prior volume. These are changes, not outlier classifications.

Investigate an index

Everything the tenant knows about one index, read-only. Pick it in the selector (every tracked index, including one without a closed day yet) or arrive from a row of Daily changes, from the Overview’s top consumers, or from a deep link:

• The index’s pool and the scopes it belongs to, with Incidents & entity settings (the entity’s own modal, where the detection and inactivity settings live), Search licensed volume and Search index events.

• Licensed in analysis period, Peak day in analysis period and, with costs on, Cost in analysis period.

• The daily-change explorer of Daily changes for this index alone: the selected day and its comparison, the index volume and change cards, the daily bars and the selected-day comparison.

• The entity’s own charts on their own windows — the daily licensed volume with its projection, the rolling volumes and their history, the outliers model and the hourly profile — as on the entity’s Volume & cost tab.

Scopes

A named scope is a set of indexes analysed together: a team, a use case, the data of Enterprise Security. Up to 50 scopes; an index may belong to several (overlaps are allowed and shown). Two sources:

• Index patterns — include and exclude patterns, * and ? as the only wildcards, case-insensitive.

• CIM datamodel macros — the scope follows the cim_<datamodel>_indexes macros, read where Enterprise Security configures them (Read macros from, per scope): this search head, the default, or a TrackMe remote account — the Enterprise Security search head tier when TrackMe runs on its own monitoring search head — as the caller, with the account’s role restrictions. A remote source that cannot be read is reported as the scope’s own error, never replaced by this search head’s macros. A macro left at its default () is unconstrained and is left out, never read as every index; a macro that is not a plain list of index terms joined by OR is unreadable and left out too. Datamodels can be excluded, and exclude patterns apply on top. When every macro is at its default the scope is empty and the tab says so.

All scopes (view=scopes) compares every enabled scope side by side: its source (with the read error when unavailable), its indexes, Licensed (period), Share of the licence or Cost (period), the last 7 days against the 7 before and the month projection — sortable, filterable by name, a row opens the scope. It is computed from the per-index search the view already runs, so there is no daily average or peak here.

A scope tab shows Licensed (period), Share of the licence (or Cost (period)), Daily average, Peak day (the peak of the scope’s own daily series, not the sum of its indexes’ peaks), Last 7d vs prior 7d and Month projection; the daily volume stacked by index with its own top consumers slider (50 by default); the scope per day with its projection; and its Indexes table.

Working days & holidays

Weekends and public holidays pull a daily average down, so the view gives a working-day average everywhere and a dedicated sub-tab: Working-day average, Weekend average, Bank holidays (the license days of the period that fall on a holiday, shown when holiday calendars apply), Holiday average, Working vs non-working, Peak working day and Quota used on a working day; the daily volume coloured by kind of day; the average per weekday (holidays left out); the top consumers and the license pools on a working day; and the holidays of the period.

A license day is a holiday when its date is covered by a holiday of the calendars in use, else a working day or a weekend day from the working weekdays (Monday to Friday by default). The holidays are either not counted (default), those of the tenant’s Bank Holidays calendars, or those of calendars you select. Only the holidays that count in a calendar are used, with their exact date every year (Easter Monday, Thanksgiving…). Working days follow the tenant’s calendars, not the four Bank Holidays effects.

Over quota

The days a license pool went over its daily quota. It needs a quota in force: the license manager’s pool size (Splunk Enterprise) or a daily quota configured per pool (any deployment, Splunk Cloud included). Without one, the tab offers Configure the daily quota.

• Window — 3, 6 (default) or 12 calendar months, ending with the last closed day.

• Tiles — In the window, This month, Last month, Rolling 60 days and Worst day.

• Licensed volume per day against the quota — one column per day, blue within the quota, red over it, with the quota line; a pool selector when several pools have a quota (the one most often over by default).

• What made the day — pick a day with the date selector or Previous day / Next day (a badge says Over quota +N% or Within quota): the pool’s 10 largest indexes that day against their usual day, the average of their previous recorded days (up to 30). Each index links to its entity. Clicking a column of the chart presets the selector.

• Days over quota — the list, with the three largest indexes of each day and their change; Per calendar month; and Splunk license warnings for context.

Note

Days over quota are not Splunk license warnings. Splunk counts one warning per pool and license-manager day over the pool’s quota. A stack under 100 GB/day has search disabled for a pool after 45 warnings in a rolling 60 days; from 100 GB/day, warnings are issued but search is not disabled. TrackMe does not know the size of the stack (a pool is a slice of it), so the view states both rules and never claims which applies. Its warning count uses the license manager’s quota only: a configured quota never raises one. Past days are measured against today’s quota.

Chargeback

The license cost of each calendar month, charged to cost centres. It needs costs (a price) and at least one centre; otherwise the tab says what to configure.

• Partition — the centres are an ordered list, each with include and exclude index patterns. An index belongs to the first centre that matches it, so every GB is charged once; the configuration shows the indexes an earlier centre already took, so you can check the order. Indexes no centre matches form the Unallocated line, or are spread across the centres in proportion to their volume.

• Statement — for the month you pick among the last 13: the Bill of the month (the calendar days of the month; the closed days so far for the current month, marked to date), Charged to the centres, Unallocated, Unused capacity or Over the licence, and Over budget. Per centre: volume, share, cost, budget used and the change against the previous month, shown only between two complete months. Each line opens on its indexes. A 13-month trend chart and Export CSV complete it.

• Prices with an effective date — a renewal does not reprice the past: each previous price applies up to the month you give it, on its own basis (annual or per GB).

• A collection gap never shrinks the bill: the days are flagged and the rules under the statement say how each basis treats them.

Important

Statements are computed live from the current cost centres and price history: no statement is frozen. To keep the statement of a closed month as it was, export it to CSV.

Configuration

The Configuration sub-tab holds five sections, saved together with Save configuration (an optional comment goes to the audit trail), with live previews as you type. The sub-tab keeps an unsaved draft while you visit the other sub-tabs; Reload saved discards it.

Section	What you set
Scopes configuration	Add, remove, enable or disable a scope; its name, description, source (Index patterns or CIM datamodel macros), include and exclude patterns, excluded datamodels. The preview lists the indexes each scope resolves to.
Working days configuration	The working weekdays and the holidays: Not counted, The tenant’s calendars or Selected calendars (with this year’s holidays of each). The preview classifies the last 30 license days.
Daily quota configuration	One row per pool with the license manager’s quota and an optional daily volume in GB. A configured quota becomes the view’s reference (or gives a Splunk Cloud stack the volume it was sized for); it never changes Splunk’s own warnings. Pools no longer reported stay listed so you can clear them.
Chargeback configuration	Enable the chargeback, choose what happens to unallocated indexes (Its own line or Spread across the centres), then the ordered cost centres (Up / Down) with a name, an owner, a monthly budget, a description and the patterns. Seed from scopes creates one centre per enabled pattern scope. The preview shows the partition.
Costs configuration	Enable the costs; the basis (Annual licence cost, with the licensed volume per day, or Cost per GB); the currency (ISO code, USD by default); the allocation (By rate or By share of the bill); the display (Next to the percentages or Instead of the percentages); and the Price history: Record a price change copies the saved price to last month before you enter the new one, Add a previous price enters an older one.

The configuration is one document per tenant with a revision: a save based on an older revision is refused, so two people editing at once never overwrite each other silently. It is backed up with the tenant. Nothing is configured by default.

Import / Export (an inner tab of Configuration, 2.4.22) moves the whole document between tenants and deployments. Export downloads the saved configuration — the five sections, the price history included, never unsaved edits — as a JSON file of format trackme.vol.license-configuration. Import reads such a file (up to 2 MiB), validates it on the server, shows the summary and the JSON, and Replace configuration replaces the destination’s whole document after a confirmation that names the tenant: no merge (an empty section removes what was there), the destination’s loaded revision must still be current, and unsaved edits or read-only access block it. Remote CIM accounts and Bank Holidays calendars are referenced by name, not exported: they must exist on the destination.

How costs are computed

Every figure uses one effective rate per GB:

• Annual licence cost, by rate — the annual cost divided by 365 and by the licensed GB per day (the configured value, else the pools’ quota in force). Each GB costs the contract rate, and the Unused capacity is the bill of the recorded days minus the cost of the volume used (negative: Over the licence).

• Annual licence cost, by share of the bill — the bill of the recorded days split across the indexes by volume, so the costs add up to the bill. It needs no licensed volume per day: the natural choice on Splunk Cloud.

• Cost per GB — that cost, always by rate.

Amounts are never rounded, so a small per-GB price never turns into zero.

Permissions

Action	Who
Open the tab and every sub-tab	Every TrackMe user who can open the tenant (trackmeuseroperations); the searches and the reads run with the user’s own session.
Change the configuration (in the page or through the AI Assistant)	trackmepoweroperations and a power or admin role on the tenant. Otherwise the Configuration sub-tab is read-only. Changes are refused while the TrackMe license is read-only.

Every saved change is audited (update license view configuration).

AI Assistant on the license view

While the Global license usage tab is open, the Tenant Home AI Assistant works on this view: it knows the view, its configuration and the figures of the sub-tab on screen (the period, the day selected in Over quota, the statement month…), offers suggested questions per sub-tab, and keeps one conversation across sub-tabs.

With your consent it can change the configuration through the Concierge: add, change, remove or reorder scopes and cost centres, set the costs, record a price change or edit the price history, set the working days, set or clear a pool’s quota, enable the chargeback. Each change is applied to the latest saved configuration, audited, and the view reloads it (an unsaved draft of your own is kept). The same permissions apply as in the page.

Known limits

• As deep as the recorded history: live collection plus backfill, up to 400 days within the retention of the license usage log. The movers need 14 recorded days, a month projection 7 closed days per index.

• Past days are measured against today’s quota (there is no quota history).

• Statements follow the current cost centres and prices (see above).

• CIM scopes read the macros of the search head TrackMe runs on.

• The periods (30 / 60 / 90 days), the over-quota windows (3 / 6 / 12 months) and the chargeback depth (13 months) are fixed choices.

Alerting

VOL entities flow through stateful alerting like every component. What is VOL-specific:

• The Create Alert wizard offers the SPLK-VOL technical alert type (a filter on object_category=splk-vol); the outliers trigger stays on by default, the outliers being the VOL state. VOL is selectable in the stateful email component filter and in the SLA breaches alert.

• Stateful alert emails embed the rolling volume charts (24 h / 12 h / 4 h / 60 min over the alert’s chart window), the outliers chart of each model, and a daily licensed volume bar chart over the last 30 license days.

• AI status reports run the ML outliers investigation for the index.

• Auto-acknowledgment, notable events, the free-style REST call, maintenance mode, CMDB and alert enrichment (splk_general_vol_cmdb_search, splk_general_vol_alert_enrichment_search) and the drilldown (TenantHome?component=vol&keyid=…) need nothing special.

See Alerting — in depth for the alert types and the email anatomy.

REST endpoints

Resource group splk_vol (browse it with API & tooling → TrackMe REST API Reference, every endpoint supports describe=true):

Endpoint	Purpose
POST /trackme/v2/splk_vol/vol_get_table	The VOL entity table of a tenant.
POST /trackme/v2/splk_vol/vol_entity_info	One index: its figures, direction, inactivity policy in force and the entity searches.
POST /trackme/v2/splk_vol/vol_get_state	The collector state (cursor, validated history, license-manager offset, deployment type) and the backfill progress.
POST /trackme/v2/splk_vol/vol_license_overview	The Global license usage context and searches (period_days 30 / 60 / 90, top_n 1 to 100, over_quota_months 3 / 6 / 12), with the license verification context: the Usage writer, the effective coverage and the per-day reconciliation receipts (also carried by vol_license_config and vol_entity_info).
POST /trackme/v2/splk_vol/vol_license_config	The license view configuration with its resolved scopes, cost-centre partition, calendars and working-days preview, and whether the caller can change it; with a draft, the draft validated section by section and previewed.
POST /trackme/v2/splk_vol/write/vol_license_config_update	Save the whole configuration (revision of the document it is based on, config, optional update_comment). A stale revision is refused (HTTP 409).
POST /trackme/v2/splk_vol/write/vol_license_config_patch	Targeted changes on the latest saved configuration (operations, 1 to 50, all applied or none): scope_upsert / scope_remove / scope_move, centre_upsert / centre_remove / centre_move, chargeback_set, costs_set, price_change, price_history_upsert / price_history_remove, working_days_set, quota_set. The AI Assistant uses this endpoint.
POST /trackme/v2/splk_vol/write/vol_set_direction	The detection direction of one or more indexes (object_list or keys_list).
POST /trackme/v2/splk_vol/write/vol_set_inactivity	The inactivity policy of one or more indexes: policy tenant / static / variable with max_sec_inactive or slots.
POST /trackme/v2/splk_vol/write/vol_backfill_control	action pause / resume / cancel / restart, optional run_now.
POST /trackme/v2/splk_vol/write/vol_bulk_edit, vol_update_priority, vol_monitoring, vol_delete, vol_update_wdays, vol_update_hours_ranges, vol_update_monitoring_time, vol_update_manual_tags, vol_update_sla_class	The generic entity operations, as for every component.
POST /trackme/v2/vtenants/admin/update_tenant_vol_inactivity_policy	The tenant inactivity policy (dispatches the tracker so the indexes apply it within a minute); read it in the vol block of GET vtenants/tenant_default_delay_config.
POST /trackme/v2/vtenants/admin/update_tenant_vol_outliers_kpis	The default outliers KPIs, with the opt-in reconcile of the existing indexes.

The tenant is created with tenant_vol_enabled (see Creating a tenant); every vol_* option below can be set in the same call.

Tenant options

Stored in the tenant’s vtenant_account and editable from Configure tenant:

Option	Default	Meaning
vol_account	local	This Splunk, or the remote account the license usage is collected from.
vol_license_search	index=_internal source="*/license_usage.log"	The license search constraint (advanced, see above).
vol_settle_seconds	300	Settle margin (60–3600 s) before a 5-minute bucket is read.
vol_late_overlap_seconds	1800	How far before its cursor each run rereads the license usage log for late events (0 disables, max 14400).
vol_backfill_days	30	Days of history replayed at creation (0 disables, max 400).
vol_default_direction	both	Default detection direction of new indexes: both / lower (drops) / upper (spikes).
vol_outliers_kpis	last_24h_volume_mb	The rolling KPIs every new index gets a model on.
splk_outliers_min_days_history	15	Minimum history before a model’s confidence is normal (the wizard’s value on a VOL tenant; the system default is 30).
vol_mloutliers_priority_filter	every priority	Priorities in scope for outliers.
vol_mloutliers_filter_expression	empty	Filter expression restricting the indexes in scope.
vol_inactive_policy	variable	The tenant inactivity policy shape.
vol_max_sec_inactive	604800	Static policy threshold (7 days).
vol_inactive_slots	working hours	Variable policy slots (Mon–Fri 08:00–19:59 at 1 day).
vol_inactive_default	604800	Variable policy threshold outside every slot (7 days).
impact_score_vol_inactive	100	Impact score of an inactive index.

The Global license usage configuration (scopes, working days, quotas, costs, chargeback) is not a tenant option: it lives in its own document, edited in the view’s Configuration sub-tab or through vol_license_config_update / vol_license_config_patch.

Metrics

Written to the tenant’s metric index under trackme.splk.vol.* (object_category=splk-vol, dimensions tenant_id, object, object_id):

Metric	Meaning	Unit
trackme.splk.vol.volume_5m_mb	Licensed volume of the 5-minute bucket (the base series; the hourly profile is mstats sum(…) span=1h).	MB
trackme.splk.vol.last_60m_volume_mb, last_4h_volume_mb, last_12h_volume_mb, last_24h_volume_mb	Rolling volume ending at the bucket end — the outliers inputs.	MB
trackme.splk.vol.last_24h_pool_pct_used	Share of the license pool over the last 24 hours (Splunk Enterprise only).	percent
trackme.splk.vol.daily_volume_mb	Licensed volume of a closed license day, stamped mid-day. Since 2.4.22 it carries the effective license weight (daily_effective_mb is written alongside as the marker; an older point without it holds the raw ingest). A point corrected by the closed-day reconciliation is stamped later in the same day with the license_day and reconciliation_revision dimensions, so latest() selects it.	MB
trackme.splk.vol.daily_ingest_mb	Raw ingest of the closed license day, before the license weight (2.4.22).	MB
trackme.splk.vol.volume_5m_ingest_mb, volume_5m_licensed_mb	The 5-minute bucket as raw ingest and as licensed volume (2.4.22; volume_5m_mb stays the raw series the outliers models train on).	MB
trackme.splk.vol.status	Decision-maker status.	status code

Every point costs 150 bytes of metric-index licensing: about 288 buckets × 5 measures per index per day live (≈ 216 KB per index per day); the backfill writes the same series over the requested depth, and the wizard shows that estimate before you confirm.

The entity searches (Open in Search on every chart) read those metrics:

| mstats latest(trackme.splk.vol.daily_volume_mb) as daily_volume_mb
    where index=<metric_index> tenant_id="<tenant>" object="<index>" span=1d
| timechart first(daily_volume_mb) as daily_volume_mb span=1d
| eval daily_volume_gb=round(daily_volume_mb/1024, 3)

Troubleshooting

Symptom	What to check
The tracker succeeds but the tenant has no entity	The license search returns no Usage row: on a custom constraint, the type / idx / b fields are not extracted (add KV_MODE = auto for the source); on a remote account, the account’s roles cannot read _internal on the remote deployment.
The daily chart shows no bar, or a bar a day early	No closed license day yet (a day sits mid-day; the entity overview chart carries its own 30-day window) — or your Splunk user’s time zone differs from the license manager’s: the daily point is placed on its license date, check the user preferences.
Every index turned red at once	Inactivity in force: the collection stopped (a failed usage search, the tracker disabled, a remote account failing) — the Manage: volume collection screen shows the cursor; the execution summary shows the run errors. Fix the source, the next run catches up 6 hours at a time.
No projection on the Volume & cost tab	Fewer than 7 closed license days, or the backfill has not merged its days yet (wait for the plan to complete).
The movers card says it needs 14 recorded days	Expected on a young tenant: the week-over-week comparison needs both weeks recorded.
A CIM scope is empty	The cim_<datamodel>_indexes macros are at their default () on the search head TrackMe runs on: configure CIM Setup there so the macros list the indexes.
Over quota is empty on Splunk Cloud	A Splunk Cloud pool has no quota: set a daily quota per pool in the Daily quota configuration section.
The Chargeback tab only explains what to configure	Costs are off, or no cost centre is defined.
Saving the configuration is refused	Someone saved since your draft started (reload, then reapply your change), you have no power or admin role on the tenant, or the TrackMe license is read-only.
Collection is held: Usage was found from multiple license writers / No Usage writer could be established	Two hosts write type=Usage into the license search (two license managers, a test stack forwarding its log to the same index), or none does yet. Scope vol_license_search with host=<license manager>; after a license manager replacement, collection resumes by itself 3 days after the old host stops writing. Nothing is lost meanwhile: the cursor waits.
The license view says the license usage events are reaching the indexers late	The collector’s window is held behind the wall clock (15 minutes or more) because the writer’s own events have not arrived; it waits rather than recording missing volume, and the hold is discounted from inactivity up to 4 hours. Check the license manager’s forwarding to the indexers and its clock.
Licensed yesterday reads Not recorded	The last closed license day has no figure yet: it closes at the license manager’s midnight and is recorded by the next collecting run, or the collection is held (see above). The last recorded day is named under the tile; an older day is never shown under that label.
About this deployment says a license day needs reconciliation attention	The day could not be reread: its RolloverSummary or Usage records are no longer searchable (retention of the license usage log, a remote account that cannot read it), the day boundaries could not be established, or the search failed. The previous figures are kept and the tracker retries every hour while the day stays in the seven-day window; the card gives the reason and the next attempt.
A day reads Source reconciled · summary differs	Not an error: TrackMe matches the detailed Usage records and Splunk’s own RolloverSummary reports a different total for that day. Both totals and the difference are on the card; the comparison keeps running daily.
Yesterday’s figure changed after the fact	The closed-day reconciliation repaired it (late Usage events, or the first reconciliation after an upgrade); the day’s revision is on the card and the corrected point carries reconciliation_revision.
A backfill plan is cancelled (live_history_reset)	The live collection re-seeded (a gap above 24 hours, a source change) while the plan was open; restart the backfill from the Manage: volume collection screen once the live cursor advances again.

See also

• VOL — Volume Outliers — the overview.

• Volume outliers on the Splunk license usage with Volume Outliers (VOL) — the end-to-end walkthrough.

• Outlier detection — in depth — the outliers engine reference.

• Alerting & third-party integration — in depth — alert types and emails.

• Metrics reference — the TrackMe metrics catalogue.