UAM — User Activity Monitoring (Beta)

Beta

User Activity Monitoring is the first Beta feature published in this documentation. It ships in TrackMe 2.4.18 as a complete, supported tenant type, and it is labelled Beta because its contracts may still move between beta releases — the finding policies, their defaults and the evidence shapes in particular. Feedback from the field is what turns a beta into a GA: tell us what you find, what is missing and what is noisy.

User Activity Monitoring answers a question none of the seven entity components asks: who is doing what on my Splunk? Which accounts run searches, from where, how often and at what cost; which scheduled searches are owned by a person who may leave tomorrow; which service account has started to run all-time searches; who is consuming the search capacity of the platform. UAM reads the sources Splunk already keeps — the audit trail (_audit), the resource introspection (_introspection) and the saved-search, user and role inventory of every search head tier — and turns them into findings: one row per policy and subject, with evidence, an investigation status, an assignee and a history.

UAM is a dedicated tenant type, not an eighth entity component: a tenant is either an entity tenant (feeds, volume, quality, workload, flex) or a UAM tenant. There is no decision maker, no impact scoring and no stateful alerting behind it — a UAM tenant is a findings-first workspace with its own trackers, its own indexed evidence, its own state-aware notifications and its own AI advisor. One UAM tenant monitors one indexing layer through one or more search head tiers: the local search head and / or remote accounts, on Splunk Enterprise or Splunk Cloud.

What UAM tracks

Signal	What it means
Scheduled searches	The inventory of every search head tier: saved searches with their owner, app, schedule, dispatch bounds and enablement; users and roles. Every change (added, changed, removed, reappeared, reclassified) is recorded as evidence, so the tenant answers who created this schedule and when.
Search activity	One audit record per search and per collection window, built from the _audit trail: the account, the app, the search id, the type and status, the run time, the events and results, the time range searched (an all-time search is flagged as such). REST job access (polling, result streams) is summarised per account instead of being indexed line by line.
Resource usage	The CPU seconds, indexer share, peak memory, elapsed time and node count of every search process, integrated from the _introspection samples of the search heads and the indexers, ad hoc and scheduled alike. The environment’s search capacity (cores and memory of the search hosts) is measured hourly and turns into the budgets of the resource policies.
Accounts and classes	Every account seen in the inventory or in the activity, with its class — human, service or system — detected from its behaviour, assigned by operator rules or set by hand. The class decides which policies apply to an account.
Findings	What the policies raise: one finding per policy and subject (a scheduled search or an account), with its evidence, a severity, an investigation status, an assignee, notes and a history; reopened by the policy when a resolved condition comes back.

How it collects

Search head tiers. A Splunk deployment has one indexing layer and one or more search head tiers — on Splunk Cloud typically the ad-hoc search head cluster and an Enterprise Security cluster; on Splunk Enterprise any number of search heads over the same indexers. Knowledge objects (saved searches, users, roles) live on each tier; the audit trail, the introspection and the internal logs of every tier are forwarded to the same indexing layer. A UAM tenant therefore selects its tiers (local and / or remote accounts) and runs the inventory once per tier, while the activity, the resources and the capacity are collected once, through the route account (the local search head when selected, else the first tier).

Trackers. Every UAM tenant is served by scheduled reports with their own lock, receipt and operational status:

Tracker	Cadence	What it does
inventory:<tier>	every 15 minutes	Reads the tier’s saved-search changes (from the write log, by exact path) and, hourly, its users and roles; diffs them against the stored rows and writes one inventory change event per difference.
listing:<tier>	once a day, off-peak	The full listing of the tier’s saved searches, the reconciliation the deltas lean on: by default between 01:00 and 06:00 in the search head’s local time (the first one within the hour of the creation). A listing costs the tier’s size whatever the filter, hence its own report, budget and window.
activity	every 5 minutes	Reads the new audit events since a durable checkpoint — never all time — window by window (5 minutes with a 60-second overlap, split when a window overflows), correlates the search lifecycle per search id, collects the resource samples of the same windows, refreshes the capacity hourly, and scans the logins and the automation evidence that drive the classification.
policies	every 15 minutes	Evaluates the inventory policies on the stored rows and the activity and resource policies on the indexed evidence of the trailing 15 minutes, reconciles the findings collection and drains the notification outbox.
backfill	hourly, self-retiring	Collects the audit and resource evidence of the past (30 days by default) in bounded buckets, newest first, until the range is covered or the sources’ retention is reached; then un-schedules itself.
AI Findings Advisor	daily window, opt-in	The scheduled review of the open findings by the AI Findings Advisor (off by default, enabled from Manage AI Agents).

Evidence. Everything the trackers observe is indexed — never kept as a growing KV snapshot — as trackme:uam:* events written through TrackMe’s dedicated trackme_uam_events.log into the tenant’s summary index (audit records, executions, resource usage, inventory changes, receipts) and notable index (finding transitions). The KV store holds the state only: the current inventory rows, the findings, the checkpoints and the receipts. Because the evidence is Splunk events, every panel of the workspace has an Open in Search and any alert, playbook or correlation search can consume it.

Checkpoint and backfill. The activity tracker never reads _audit all time: it starts at now minus one window on the first run and advances a durable cursor after every window it published, so a run that dies is replayed, never skipped, and a degraded source (a search peer answering every search with a warning) is retried a bounded number of times before the window is recorded as a gap rather than stalling the cursor for good. The backfill covers the history before the live start — 30 days by default, bounded by the retention of _audit (and of _introspection, typically shorter) — hour after hour, without ever collecting a span twice. Progress and controls live in the Collection health tab; see Backfill.

Accounts and classification

A finding about an account only makes sense once you know what the account is. UAM keeps three classes:

• human — a person. Detected automatically from interactive activity: a search run from Splunk Web (Splunk’s UI:* provenance — the Search app, a dashboard, a report) or a browser login to Splunk Web. One interactive search or login is enough, and the class is sticky (a person on leave stays a person).

• service — automation. Detected after 7 distinct UTC days of automation-only activity (REST jobs, the scheduler, data-model acceleration) and nothing a person does; withdrawn as soon as an ambiguous search appears (a labelled Splunk Web dispatch, a splunkjs page, an MCP client…). Two limits of the beta: the automation scan reads the legacy key=value audit format — a deployment whose _audit is JSON-only gets no automatic service detection — and the seven days must lie inside the audit retention.

• system — the Splunk context identities (nobody, splunk-system-user) and the built-in admin, seeded by default. System accounts are never evaluated by the account-scoped policies.

On top of the detection, the administrator declares classification rules (exact or wildcard patterns, ordered, no regular expressions) and manual classes per account, from the Users tab or from Settings → Account classification, with a preview before saving. Precedence, always: manual class, then the system defaults, then the first matching rule, then the human detection, then the service detection. An account nothing classifies stays unclassified — a governance gap the unclassified_schedule policy can surface — and is skipped by the policies scoped to human accounts.

Tip

Classify the monitoring itself first. Give the UAM tenant a dedicated service account with administrative rights as its owner (svc-trackme, for instance), and classify that account and your administrator accounts as system: they run the trackers, the acceleration and the platform’s own housekeeping, and must never appear in the findings. The trackme and Splunk_SA_CIM apps are excluded from every policy by default for the same reason.

Findings and policies

Findings come from a catalogue of eleven policies in three families. Six are enabled on a new tenant; the others are opt-in. Every policy has a severity, exact owner / account and app exclusions, and its own thresholds; the two resource policies derive their thresholds from the measured capacity of the environment (auto-thresholds), so nobody maintains absolute CPU or memory figures.

Policy	Family	Default	What it detects
personal_schedule	inventory	on, low	An enabled scheduled search owned by an account classified as human — a schedule that dies with its owner’s account. Low by design: common, and it must not dominate the workspace.
unclassified_schedule	inventory	off	An enabled scheduled search whose owner has no human / service classification: a governance gap, not misuse.
dense_schedule	inventory	off	A cron expression with more than N distinct minute slots in an active hour (12 by default) — configured frequency, not observed launches.
all_time_schedule	inventory	on	A scheduled search whose dispatch earliest time is 0 or empty (Splunk’s default: no lower bound).
realtime_schedule	inventory	on	A scheduled search dispatched with a real-time time range (rt, rt-5m@m…).
launch_burst	activity	off	An account that launched more than N distinct searches in the policy window (200 by default).
costly_search	activity	off	An account that ran more than N searches (3) whose reported run time reached the minimum (300 s).
all_time_search	activity	on, high	An account that ran an all-time search (earliest bound 0 / N/A) — more than 0 by default: the one thing that should not happen.
resource_consumption	resource	on, auto	An account whose CPU seconds over the window exceed its allowance — 10 % of the search CPU capacity per window by default. Its share of the environment’s search CPU is evidence; above 40 % the severity is raised one level.
resource_intensity	resource	on, auto	An account that ran more than N (3) resource-intensive searches: a search whose CPU reached 2 % of the search capacity per window or whose peak memory reached 10 % of the smallest search host.
resource_deviation	resource	off, preview	An account whose CPU today is unusual for its daily baseline (a multiple of its p95 over 14 days, after 7 fully covered days). Preview-only in the beta: it records matches without creating findings.

Severity is medium unless stated. The resource policies stay in a learning state until the tenant holds 24 hours of resource history and a capacity record — nothing is evaluated, nothing is closed — and the Policies tab says why. A finding carries its evidence (the window, the counts, the thresholds in force and their source, the top apps or searches), the policy’s suggested remediation, an investigation status (open, investigating, resolved, dismissed), an assignee, notes and a history of every decision. Resolve concludes an investigation and lets the policy reopen the finding if the condition comes back; dismiss accepts it — permanently or for a duration, after which the policy brings it back for another look. See Findings lifecycle.

The workspace

A UAM tenant opens its own Tenant Home, organised in eight tabs. Every tile and panel carries the app-wide menu (Open in Search, Refresh), the KPI tiles drill into the table filters, and a Collect now button dispatches the trackers on demand.

Findings

The investigation backlog: tiles (open, high / critical, new in the range, assigned to me, resolved, dismissed), a transitions timeline, and the findings table with its filters (status, severity, policy, subject, assignee, text) and time range. A row opens the Investigate view — the subject, the evidence, the suggested remediation, the assignment, the history, Open record / events in search, the notes and the AI Findings Advisor — and the row menu or the Bulk edit button records decisions on one or many findings.

Users

The account-centric view across every tier: accounts in range, inventoried users, classified and unclassified counts; one row per account with its class (and the detected marker with its evidence), its roles, its tiers, its findings, its audit records, executions, last activity and apps. A click opens the account deep-dive: the profile, its findings, an executions timeline by search type, the most recent searches (with the SPL when it is stored), the scheduled searches it owns and its resource usage — CPU, indexer share, peak memory, share of the environment and the most expensive searches. The bulk edit classifies a selection or decides on their open findings.

Activity and Scheduled searches

Activity is the raw evidence: a timeline of the audit records over the selected range, split by account source, evidence status or account, and the records themselves (account, app, search type, search id, outcome, run time, evidence status), each one inspectable. Scheduled searches is the inventory: tiles for the scheduled searches, the full listings, the last inventory changes and the users and roles; the table of the saved searches with owner, owner class, schedule, enablement, first and last seen, per tier. The per-viewer switch Exclude TrackMe activity leaves the trackme app out of the activity views.

Policies

The catalogue with, per policy, its family, enablement, severity, thresholds (the effective values when they are derived from the capacity), exclusions and the counters of the last evaluation. The policy modal edits one policy and previews the draft against the current inventory before saving; the policy insight explains an evaluation — the verdict, the population, why records were unknown, excluded or not applicable, the owners behind the unknowns (classifiable in place), the exclusions, the run and a seven-day history.

Collection health

The operational view: the search head tiers and the route, every tracker with its last run (status, duration, coverage, events, diagnostics) and its seven-day history, the activity checkpoint and its lag, the inventory datasets per tier, the audit windows of the last run, the measured capacity, and the historical backfill with its progress, edges and controls (pause, resume, cancel, plan again, run now). A stopped run whose lock survived is recovered from here.

Settings

Six categories with a shared draft and one Save: Environment (the tiers and the route), Collection (scheduled trackers, Store SPL search, resource collection, source version assumption), Indexes (per evidence family), Collection limits (activity, resources, inventory, policies and backfill budgets), Notifications and Account classification (manual classes, rules, detection switches, preview).

Audit changes

The shared TrackMe audit of the tenant — every configuration change, decision and collection dispatched by a user, with who did what and when — the same view every tenant type offers.

Creating a UAM tenant

1. Create a tracking tenant and pick the User Activity Monitoring card (badge Beta, the sixth card of the chooser).

2. Basics — the tenant identifier, alias and description.

3. Configure UAM — the search head tiers (the local search head and / or remote accounts; every remote tier passes its connectivity test before the creation), the route account derived from them, and the historical backfill (30 days by default, from none to 90).

4. Indexes & RBAC — the TrackMe indexes, the roles and the owner: prefer a dedicated service account with administrative rights (see the tip above).

5. Review and create. The trackers are scheduled immediately; the first inventory, activity and policies runs follow within minutes, and the backfill plans itself at its first hourly slot.

The wizard is walked step by step in Creating a splk-uam tenant, and end to end on a live deployment in the User Activity Monitoring white paper.

Note

Editions. UAM is available on the Enterprise and Unlimited editions, their trials and the Developer license; it is not available on the Foundation edition or the Foundation trial. The evidence UAM indexes counts against your Splunk license like any TrackMe summary data — see The evidence for the footprint and the Store SPL search switch (off by default: the SPL of the reported searches can carry literals and secrets, and is only indexed when you switch it on).

See also

• User Activity Monitoring — in depth — the tenant model, the tiers, the trackers, the evidence, identity and classification, the policy catalogue, the findings lifecycle, notifications, operations, REST and the tenant options.

• Who is doing what on your Splunk? — the white paper: an end-to-end walkthrough on a live Splunk Cloud deployment.

• Configuration Guardian — the uam_collection_health check.

• The AI Advisors — the AI advisors, including the scheduled review of findings.