User Activity Monitoring¶
TrackMe has always watched the data. User Activity Monitoring watches the other half of a Splunk platform: the people and the automation that use it. It answers a question none of the entity components asks — who is doing what on my Splunk? Which accounts run searches, how often and at what cost; which scheduled searches are owned by a person who may leave tomorrow; which account started to run all-time searches; who consumes the search capacity of the platform; whose CSV lookups keep growing; which indexes are read, by whom, and which receive data no one reads.
A User Activity Monitoring tenant (UAM) reads what Splunk already keeps — the audit trail, the search resource records and the knowledge objects of every search head — and turns it into findings: one finding per policy and subject, with its evidence, a severity, an investigation status, an assignee, notes and a history. You investigate, decide, notify the people concerned and move on.
UAM is a dedicated tenant type, not an eighth entity component: a tenant is either an entity tenant (DSM, DHM, MHM, FLX, FQM, WLK, VOL) or a UAM tenant, never both. There is no decision maker, no impact score and no state colour behind it: what you work with are findings. UAM shares its findings model — Finding groups, triage, assignment, notifications, findings actions, the AI Agent — with the other dedicated tenant types (Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream and Cribl Edge); those concepts are explained once in Findings, Finding groups and findings actions, and this page documents how they work in UAM.
Tip
This page is the task-oriented guide: set the tenant up, find your way in the workspace, triage, communicate and tune. Every setting with its default and bounds, the schedules, the indexed evidence, the REST API and the troubleshooting table are in User Activity Monitoring — reference.
What UAM watches¶
One UAM tenant monitors one indexing layer seen through one to ten search head tiers: the local search head and / or remote Splunk deployments reached through TrackMe remote accounts, on Splunk Enterprise or Splunk Cloud.
What |
What UAM knows about it |
|---|---|
Accounts |
Every account seen in the users inventory or in the search activity — people, service accounts and Splunk’s own system identities — with its class (human, service or system), its roles, the tiers it exists on and its email address when Splunk has one. |
Search activity |
One compact audit record per search and collection window, read from |
Resource usage |
The CPU time and peak memory of every search, on the search heads and on the indexers, and each account’s share of the environment’s search capacity. CPU follows Splunk’s own accounting where it exists. |
Saved searches |
The saved-search inventory of every tier: owner, app, schedule, dispatch time range, enablement, and every change (added, changed, removed). |
Other knowledge objects |
Dashboards, macros, event types, field extractions and transforms, field aliases and calculated fields, lookup definitions, automatic lookups, CSV lookup files (size, owner, modification time — never their contents) and tags, read from the Splunk REST API of each tier. |
Data usage |
The indexes every search names, by account, app and path (literal term, macro, event type, saved search, data model or wildcard), the knowledge objects that declare each index, and — through linked VOL and DSM tenants — what each index ingests and costs. See Data usage. |
Findings |
What the policies raise: sixteen built-in policies and the custom policies you write in SPL. |
Note
What UAM claims, and what it does not.
An account in UAM is the account reported by Splunk: it is not a verified person. Shared credentials or an integration running as a person’s account look like that person.
Collection is bounded by time only, and nothing is dropped to fit a row count: what a run cannot finish in its time continues at the next run. Coverage still depends on what the Splunk sources retain and what the collection accounts can see; gaps and partial coverage are recorded and shown in Collection health, never hidden.
There is no machine learning in UAM: thresholds are explicit, and the daily CPU baseline is a percentile over the account’s recent days.
UAM never changes, dispatches or deletes anything it observes. A decision on a finding stays in TrackMe.
Before you start¶
A licence beyond Foundation. UAM is available with the Enterprise and Unlimited editions, the Unlimited trial and the Developer licence. It is not available on the Foundation edition or the Foundation trial. When a licence expires, the workspace stays readable while changes are refused, and scheduled collection continues until the licence enforcement ladder stops the tenants.
The Splunk sources. UAM reads _audit (search activity), _introspection (search
resource records and capacity) and the indexers’ search completion records (CPU accounting),
through one search head tier called the route account. Splunk Enterprise 9.4, 10.4 and
10.6 and Splunk Cloud 10.5 are qualified; other versions run as well and only add a soft
diagnostic in Collection health. Three things to know:
The automatic detection of service accounts relies on the classic
key=valueformat of the_auditlines. A deployment whose_auditonly contains the JSON format gets no automatic service detection (human detection and your own rules still work).What UAM can look back on is bounded by the retention of these indexes.
_introspectionis often kept for a shorter time than_audit, so resource history may end earlier than audit history.Before Splunk 10.4, the lookup file listing gives no file size: UAM reads the sizes of the CSV lookup files through Splunk’s file explorer endpoint (
admin/file-explorer), so the collection account of each tier must be allowed to read it. By default only theadminandsplunk-system-roleroles can, so a least-privilege account cannot retrieve CSV lookup sizes before Splunk 10.4. When the read is refused, Collection health reports Lookup sizes refused and the CSV lookup footprint policy cannot judge that tier. From Splunk 10.4, the listing reports the size and no extra access is needed.
Remote tiers. Each additional search head tier is a TrackMe remote account (Configuration → Remote accounts, see Remote Splunk deployments). The wizard tests each one. Live reads you make from the workspace as yourself — Fetch SPL, Fetch definition — also need your roles to be allowed on that remote account.
A tenant owner. Scheduled collection runs as the tenant owner. Prefer a dedicated
service account (svc-a in the examples of this page) over nobody: the scheduled UAM
AI Agent cannot run as nobody or splunk-system-user.
The collection accounts and their capabilities. Each search head tier is collected as one
account: the tenant owner on the local search head, the token account of its remote
account on a remote tier. UAM needs no list_settings and no list_storage_passwords.
TrackMe checks these accounts when the tenant is created and on every health cycle. A gap
raises the Configuration Guardian alert UAM collection permissions need attention, which
names the tier and what is missing (Configuration Guardian).
Capability |
Local tier (tenant owner) |
Remote tier (token account) |
What it is for |
|---|---|---|---|
|
Required |
Required |
Run the collection searches. |
|
Required, with one of the tenant’s admin roles |
Required |
The UAM administrative boundary. On the local tier, users and roles are read through
TrackMe’s identity reader, which serves the tenant’s administrators only. The
capability is defined by the TrackMe app: on a remote deployment where TrackMe is not
installed, only |
|
Not required |
Required |
List every account and role of the tier. Without them, Splunk lists only the token account and its own roles, and the tier’s users and roles catalogs hold one account. On the local tier, TrackMe reads them with its system token for the tenant owner. |
|
On a search head cluster |
On a search head cluster |
Read the cluster members, to attribute activity to the tier. |
Beyond capabilities:
The route account must be allowed to search
_audit,_introspectionand_internal(the indexers’ search completion records, and the searches they received, which Data usage reads).Before Splunk 10.4, the collection account of each tier reads
admin/file-explorerfor the lookup sizes (see above).Index permissions, knowledge-object permissions and the search scope of each account still decide what UAM sees. The check confirms the capabilities, not complete visibility: Collection health reports Users or roles listed as one account sees them (
identity_visibility_limited) when a tier’s listing still holds only the reading account.
Who can use it. UAM evidence is audit data, so the workspace is for the TrackMe administrators of the tenant — accounts holding the TrackMe administrator capability and one of the tenant’s admin roles; Splunk administrators can do everything on every tenant. Power users of the tenant have read access through the REST API (and can send user communications and run findings actions through it); read-only users have no UAM access. The workspace’s own searches (the Data usage views, the Privileged users activity, Fetch SPL) run as you, so your roles also need the tenant’s TrackMe indexes, and the search text index when it is a restricted one.
Email delivery (optional). Notifications and communications are sent through a TrackMe email delivery account (Configuration → Emails), or the local mail transfer agent when none is selected. The account’s allowed-domain rules apply.
Note
UAM indexes its evidence in the tenant’s summary index and its finding events in the
tenant’s notable index, as trackme:uam:* events. This evidence counts against your
Splunk licence like any TrackMe summary data; it is compact by design (one record per
search and window). Some indexed fields of these events take effect after the splunkd
restart that follows the app installation or upgrade; until then, reads fall back to
slower searches.
Create the tenant¶
From the Virtual Tenants page, open Actions → Create a tracking tenant and choose the User Activity Monitoring card (Investigate Splunk user activity and policy findings). The wizard has four steps.
Step 1 — Basics¶
The tenant identifier (lowercase letters, digits and hyphens, up to 20 characters; it cannot change later), an optional alias shown on the tenant card and an optional description.
Step 2 — Configure UAM¶
Search head tiers. Local search head is selected by default; each TrackMe remote account can be added as another tier, up to ten. Select every search head tier that shares the indexing layer you want to monitor — for example an ad hoc search head cluster and an Enterprise Security cluster on the same indexers. Remote accounts are tested as soon as you select them, and creation waits until every selected one is reachable (Retry connection test if one fails).
Route account. Derived from your selection: the local search head when it is selected, otherwise the first remote tier. The indexing-layer searches — audit activity, resource usage, capacity, logins and automation — go through it, once for the whole layer, while the knowledge objects are read on every tier.
One indexing layer per tenant. All the tiers of a tenant must search the same indexers. Monitoring two independent deployments takes two tenants.
Historical backfill. How far back UAM collects the audit and resource evidence that existed before the tenant: No backfill (the default), 7, 14, 30, 60 or 90 days. The backfill runs hourly in 5-minute buckets, newest history first, and retires itself once the range is covered or the retention of the sources is reached. It makes the account classification and the activity history useful from the first day, at the cost of reading that history from the indexers: on a busy deployment, start without it, or with a short range, and extend it later in Settings → Collection limits.
Activity scope (advanced). Optional hostname patterns (* and ?, up to 20
include and 20 exclude patterns) that restrict which search heads’ activity is collected,
for example to leave out a test search head that shares the indexers. Exclusions win. The
scope is part of the base searches, so the indexers only return the events of the search heads
in scope: on a large indexing layer shared by several search head clusters, it is also the
first way to make collection lighter.
Step 3 — Indexes & RBAC¶
The TrackMe indexes of the tenant (summary, audit, metric and notable), the tenant owner and the admin, power and user roles. The summary index receives the evidence and the notable index the finding events; each family can be pointed at another index later in Settings → Indexes.
Step 4 — Review¶
Check the summary — the tiers with the route, the owner, the activity scope, the backfill — and create. Everything is provisioned and scheduled at once: the inventory, audit activity and resource usage reports start within minutes, the account signals and the policies follow, and each tier’s first full saved-search listing runs within the hour. If the creation result is incomplete, the wizard says so and the workspace offers Repair UAM workspace.
The step-by-step creation is also described in Creating a splk-uam tenant.
After creation¶
The tenant appears on the Virtual Tenants page with its open Finding groups and findings, the high or critical ones, and its collection status — here the historical backfill in progress. Selecting it opens the workspace; double-clicking it opens a read-only overview.
Your first hour with a new tenant:
Open Collection health: the Status view says whether anything needs attention, and Trackers shows the first inventory, audit activity, resource usage, account signals and policies runs of every tier.
Classify the monitoring accounts as system — the tenant owner, your administrator accounts — so that they never appear in findings (see Accounts and classification).
Review the Policies tab: the enabled policies, their severities and exclusions.
Come back to Findings after the next policies run (every 15 minutes).
The workspace¶
The workspace opens on the Findings tab. At the top, the status strip tells you that the tenant is working: the search head tiers (the route account marked · route), the age of the last Inventory, Audit activity, Resource usage, Account signals, Data usage and Policies runs, the activity checkpoint lag and coverage, the backfill progress, a Configuration changed since last run warning after a settings change, Refresh, and Collect now — which runs every tier’s inventory, then the audit activity, the data usage, the resource usage and the account signals, then the policies, with the current configuration, and every tier’s full saved-search listing alongside.
Eleven tabs follow: Findings, Users, Privileged users, Data usage, Activity, Scheduled searches, Policies, Audit changes, Collection health, AI Agent and Settings.
A few conventions apply everywhere, shared with the other findings workspaces (The shared workspace conventions):
Column headers filter and sort the loaded page; the toolbar filters search everything.
Every panel has a ⋮ menu with Open in Search (the search behind the figure, ready to edit) and Refresh.
Three personal display preferences per tenant live in the Tenant Home settings panel, under Personal preferences for this tenant: Collection monitors (the status strip), Single stats and Charts. They change your view only.
Exclude TrackMe activity (Users, Activity and the account view) leaves out the searches run in the
trackmeapp — TrackMe’s own scheduled trackers (recognised by their search id) and your own use of TrackMe. An ad-hoc search that reports another app or none, such as the tracker searches TrackMe dispatches over REST, cannot be attributed and stays in. Findings are not affected. It is remembered per viewer.Notification emails link straight to a finding or a Finding group, opened in its investigation.
Findings¶
Your investigation backlog. Tiles count Open groups, High / critical groups, Groups assigned to me, Resolved groups and Dismissed groups — Finding groups — plus New findings in range (findings first seen in the time range), and drive the filters; charts show the finding transitions over the time range (24 hours by default) and the open findings by severity and by policy. Filters cover the status, severity, policy, subject (accounts, saved searches or roles), first seen, assignment (including assignment groups) and free text (owner, app, search name).
The table lists one row per Finding group: an account and the saved searches it owns. A group of several findings has an expander showing each member with its own policy, severity and whether the latest policies run still observed it. Selecting rows offers Merge into one Finding group, Add to the Finding group (a manual group), Run action, Assign to… and Bulk edit; AI Agent review asks the AI Agent to review the open and investigating findings.
Selecting a row opens the Investigate view: the subject, the findings of the Finding group, What happened in plain words, the suggested remediation, the evidence of the selected finding, Triage of the Finding group (status, dismissal, assignee, notification snooze, severity, investigation note), the notes and both histories — the finding’s and the group’s. From there, open the account view or the saved search view, or the record and its events in Search.
Each finding explains its own evidence. A CSV lookup footprint finding, for instance, lists the files attributed to the account, their size and how each was attributed: to its probable writer — the account whose searches wrote the file — or, when no writer could be identified, to its recorded owner.
Users and the account view¶
Every account across the tiers: tiles for the accounts in range, inventoried, classified and unclassified; filters by class, source and activity range; columns for email addresses, class (with a detected marker when a detection set it), roles, tiers, findings, audit records, executions, last seen and apps. The row menu sets the class or overrides the email address used for user notifications (Override email address…); a selection can be classified at once, or decided on as a whole (resolve, dismiss or mark investigating its open findings).
Selecting an account opens the account view:
Activity & usage — executions and run time over time by search type, CPU over time, the daily CPU baseline, the resource usage (CPU time, peak memory, share of the environment), the most expensive and the most recent searches with Fetch SPL, the account’s findings and the scheduled searches it owns.
Data accesses — which indexes the account read, over time: a chart of its named reads per index and one lane per index with a marker for each observation (see Data usage).
The account’s catalogs, one tab per family: Saved searches & alerts, Dashboards, Macros, Event types, Data models, Indexes, Fields, Lookups and Tags — what this account owns on each tier.
The resource usage tells what an account costs: here a service account whose scheduled detection searches use 0.2% of the environment’s search CPU over the day, with the most expensive runs listed first. Searches the resource records could not measure are counted and shown as such, never estimated.
Fetch SPL shows the text of a search. By default UAM does not store search text: it reads
it live from _audit, as you, and stores nothing. If you switch on Store SPL search
(Settings → Collection, off by default), the text is indexed in its own record in the
search-text index — point that index at a restricted index first, since search text can
contain literals and secrets. When a text cannot be shown, the reason is given.
Search knowledge objects (on the Users tab) searches the collected catalogs across every
owner: a name or a wildcard pattern, filtered by object type, tier, app, owner and sharing.
Use it to answer “who owns the macro app_a_base?” or “which dashboards use this name on
which tier?”.
Note
The catalogs reflect what the collection account can see on each tier: the coverage of private objects depends on its permissions. They are collected on a cadence (daily in the off-peak window by default for most families, every 15 minutes for CSV lookup files, hourly for users and roles), not as atomic snapshots. A listing too long for one run continues at the next one, and the previous catalog stays in use until the new one is complete.
Privileged users¶
Which accounts hold high privileges on the monitored search head tiers, and why. An account is privileged when one of its roles — or a role that role imports, however indirectly — is a privileged role or grants a privileged capability:
Privileged roles: every role whose name matches a pattern (
*admin*by default, case-insensitive,*and?), plus the roles you add by name, minus the roles you exclude. The patterns are live: a role created later is privileged as soon as its name matches.trackme_adminis excluded by default: TrackMe’s own administrator role holds TrackMe capabilities, not Splunk-level ones.Privileged capabilities:
admin_all_objects,change_authentication,delete_by_keyword,edit_roles,edit_tokens_all,edit_user,list_settings,list_storage_passwordsandrest_apps_managementby default, granted directly or through an imported role.An exclusion only removes a role’s name rule: an excluded role that grants a privileged capability still makes its users privileged, through the capability.
The tab shows the privileged accounts and their share of all accounts, the privileged roles,
the privileged capabilities in use and the privileged accounts active in the last 24 hours;
charts of why accounts are privileged (role, capability or both), of their classes and of the
accounts holding each privileged capability; the privileged accounts with their reasons
in words (team_lead imports admin (pattern *admin*), secops grants
list_storage_passwords), last seen and executions; and the roles of every tier, the
privileged ones by default. An account present on several tiers is one account, each reason
naming the tier that grants it.
Everything is read from the users and roles catalogs UAM already collects: the view follows their cadence (hourly by default) and makes no call to Splunk. The rules are set in Settings → Privileged users: the patterns, and two dual lists — the roles and the capabilities the tiers hold — where moving a role a pattern matches out of the privileged list excludes it, and moving any other role in adds it by name.
Splunk grants roles and capabilities through direct membership and through inheritance, and a role inherits from roles that inherit further: every reason keeps its chain — team_lead imports admin, team_lead inherits admin through ops, leaf inherits list_settings from base through mid. The full rules are in Privileged users.
Four privileged policies build on this assessment, all off by default: an account that became privileged, a role that gained a privileged capability, a dormant privileged account, and privileged accounts that differ from a reference you maintain. See Privileged policies.
Data usage¶
Which indexes are read, by whom, and which receive data no one reads? UAM already reads
every search from the audit trail. The Data usage tab turns those searches into a map of
your data: for every index, the searches that named it, the accounts and apps behind them, the
path they took (a literal index= term, a macro, an event type, a saved search, a data model
or a wildcard), the knowledge objects that declare it, and — when the tenant is linked to a VOL
or DSM tenant — what the index ingests per day and what it costs.
It rests on one rule: an absence of reads is a claim, made only when the evidence supports it. An index is never called unread from a few hours of collection, an incomplete read or a wildcard that may reach it. Every view says what it covers, in one coverage card under its controls: when the collection started and what that means for the view (“Collected since 27 Oct 2026 (11.8 % of the selected period). “Read by no one” is provisional until 3 Nov 2026.”), a fault with its cause, or Coverage complete.
Five views share the same controls: the search head tier, internal indexes (left out by default), the scope (TrackMe’s own searches and the excluded apps, set in Settings → Data usage) and the time range (the last 7 days by default).
Every tile, chart and table of the Overview, Value & cost, Knowledge objects and Evidence views has a three-dots menu with Open in Search. It opens the search over the recorded facts behind that view, with a note on what only the map adds when you read the view: the catalog expansion of wildcards, the role scopes and the VOL volumes. How each kind of claim is recorded is described in Data usage reference: how each claim is processed.
Overview — where the data goes. One sentence answers the question: Of 6.60 GB/day ingested, 1.73 GB/day is read by name, 1.52 GB/day only through wildcards and 0.00 GB/day by no one so far. A bar splits the volume by usage, and the hero shows it either as a flow — from data families (indexes sharing a name prefix) to their usage, then to the accounts that read them — or as a treemap of ingestion, one tile per index sized by its daily volume and coloured by its usage.
The usage classes are decided across the whole indexing layer, never per tier: Read by
name (a search named the index), Wildcard only (only wildcards such as index=cribl_*
reach it: they do not prove which index supplied the events), No reader, Excluded apps
only (only the apps you exclude read it, for example trackme) and TrackMe’s own (the
indexes TrackMe writes to, never a review candidate). An index without a linked volume is
counted but not weighed.
Under the hero, three panels list the index references by path, the index usage verdicts and the reference paths (literal, macro, event type, saved search, data model).
Value & cost — weigh what an index costs against how it is read. The treemap of ingestion by usage sits beside Volume against reads (log scales, one dot per index: the shaded corner holds the indexes larger than the median and read fewer than ten times), and Cost of disuse states what the indexes no one accessed, and those only wildcards reach, cost per day.
The Decisions list turns it into work: one row per index, largest first, with its daily volume and cost, its usage, who read it, its reads per GB, the knowledge objects that declare it and a suggested step (“Only wildcard searches reach it: confirm those searches need this data, or scope them by name”). Five views filter it: To review (no reader, wildcard only, excluded apps only and the least accessed), No reader, Wildcard only, Least accessed (the quarter of the indexes read by name that are read the least for their volume; the ranking needs at least four such indexes with a daily volume) and Every index. An empty list says why — for example, while the collection is younger than the period, nothing is called unread. The list pages 10, 25, 50 or 100 rows.
Index activity — investigate an index. One row per index and tier: the incoming data (from
the linked VOL or DSM tenant), the usage verdict, the source reads, the reads the indexers
received (on distributed deployments, the indexers’ own record of the searches they ran,
shown beside the search head’s figures, never added to them), the wildcard references, the
write references (collect destinations, and the target index of a saved search’s summary
indexing or Log event alert action), who read it, from which apps, when, and through which
paths. Filters cover the usage, the path (its wildcard choice selects the wildcard
references), the account class, the index name, the account and the app.
Detail opens the index. Source references gives the tiles — source reads, indirect and write references, the sourcetypes reported (best effort: a sourcetype never gets a verdict of its own), the volume per day and its cost, the reads the indexers received and the knowledge objects that declare it — and one row per account, app, sourcetype and path, with Fetch SPL examples to read the retained searches. Access timeline charts the named reads of the index over the period, clock-aligned, from the start of the collection.
Knowledge objects — declared against observed. Each published saved search, macro, event type, dashboard and data model, with the indexes its definition declares (wildcards expanded across the index catalog, shown muted) against the indexes its searches read in the period (an index read but not declared is outlined). Tiles count the states and filter the list:
In use — ran in the period and read the indexes its definition declares.
Ran without declaring — ran, but its definition names no index: what it read was resolved at run time.
Not run — declares indexes and did not run in the period, dated from when saved search runs began to be recorded.
Not due in this period — a scheduled search whose schedule leaves a longer gap than the period covers: it was not due, so it is not called Not run.
Disabled — disabled in its configuration.
Building block · used / not seen — a macro, event type or data model that searches went through, or that no observed search resolved directly.
Dashboard · viewed / not viewed — a dashboard whose panels ran searches in the period, or not.
An object opens its detail: what it declares, what it read, how many searches ran and when.
Evidence — what supports these numbers. The collection behind the figures: the fact records read, the collection runs and their outcome, the index catalog of each tier, the indexers’ confirmations, and the searches whose indexes could not be resolved, with the reason for each (a macro or saved search that cannot be found, a dynamic term, a wildcard role default) and the steps to resolve it.
An account’s data accesses. The account view has a Data accesses tab: the named reads of the account per index over the period, and Which data this account used, one lane per index with a marker for each group of nearby observations. Select a marker for the app, the usage origin and the claims behind it. An index view of the same timeline shows who used this index.
Searches run as long as they need. Every view reads the evidence through your own Splunk search jobs, started when the view opens and followed by the UI: the loader names what is still being read (Still reading Collection runs…), and the searches are cancelled when you change the selection or close the view. Nothing is cut after a fixed time: a long period or a large tenant takes longer, and is complete when it is shown. Only a safety timeout of 10 minutes applies.
Settings → Data usage. Each search head tier is linked to the VOL tenant that gives its
index volumes and to the DSM tenants that give its incoming data. Matching is automatic by
default — the first VOL tenant and every DSM tenant you can see that is enabled and has the
component, whatever account it collects through (2.4.23) — and you can override or remove a
link. Indexes are matched by name: unselect a tenant that measures other indexers. A link grants no access: every read repeats the visibility check of the caller. The same
section sets the apps whose reads are left out (trackme and Splunk_SA_CIM by default),
whether internal indexes are shown, and the collection’s own limits.
Note
What Data usage claims, and what it does not.
A read is a search that named an index — directly or through a macro, event type, saved search or data model that resolves to it. A wildcard is a reference, not a read:
index=cribl_*does not prove which index supplied the events.Each search is counted once. A search Splunk refused at its quota is not a read; a canceled or failed search that started is.
A sourcetype is reported on a best-effort basis and never gets a verdict.
The map covers what the audit trail of the indexing layer records and what the collection has seen so far. Until the collection covers the period, or seven days of it, nothing is called unread.
A declaration in a knowledge object is not a read, and a link to a VOL or DSM tenant is not evidence of a search.
The collection behind the tab — the Data usage report, its facts, the resolution limits and the REST operations — is described in Data usage.
How each kind of claim is processed (named reads, wildcards and their narrowing to the reader’s roles, searches the roles deny, definitions, index-less searches, late filters, writes, indexer confirmations, declarations, incoming data, verdicts and coverage), with diagrams, worked examples and limits, is the Data usage reference: how each claim is processed. The same reference opens in the product from every How this works link of the tab.
Activity¶
The raw evidence: Audit records or Correlated executions over a time range, a timeline split by account source, evidence status or account, account and app filters, and the 500 most recent records with an evidence inspector (Inspect). Executions are correlated from the audit records when you read them — the start and the completion of each search, the accounts and apps reported — and a notice says when an answer covers part of the range only.
Scheduled searches and the saved search view¶
The saved-search inventory per tier: tiles for the scheduled searches, the last full listing, the changes of the last run and the users and roles counts; the table with owner, owner class, schedule, enablement and first and last seen. Selecting a name opens the saved search view: its definition from the inventory, Fetch definition for the live definition, its runs with run time, CPU and peak memory, its findings and its change history, and a way to the owner’s account view.
Peak memory can read not measured for short searches that Splunk’s resource records never sampled. A saved search with the same name, app and owner on two tiers shares one set of run figures.
Policies¶
The built-in catalogue grouped by family (inventory, activity, resource, privileged) and your custom policies, shown as Cards, Compact cards or a List (remembered per user). Each policy shows its status, severity, account scope, thresholds (the effective values when they are derived from the capacity), exclusions, what it does on bank holidays and its last evaluation. Selecting the last evaluation opens the policy insight; Edit opens the policy editor. See Tuning the policies.
Audit changes¶
TrackMe’s own audit of this tenant: every configuration change, decision and collection request, with who did what and when — the same view every tenant type offers.
Collection health¶
Is UAM collecting what it should, and if not, why and what to do? Everything that says something is wrong comes with its explanation, its evidence and its action. Four views:
Status — one row per tracker that needs attention, from the same assessment the Configuration Guardian raises, so the page and the alert never disagree. A row expands to every reason with its meaning and where it was reported, What to do, the facts of the last run (status, finish, consecutive failures, search job, report and its live schedule, lock), the policies a run could not complete (with Show why), shortcuts to the evidence (Receipts, Tracker log, Search job, Report, Scheduler log) and Run now / Recover. Below, Problems in the last 24 hours lists the failed and incomplete runs of each tracker, recovered or not. Flow draws UAM’s own collection pipeline — the search head tiers and the indexing layer, the trackers, what they store and the policies — coloured by health.
Trackers — every tracker in one table: status, live schedule (read from Splunk), last run, last collected, duration, diagnostics, Run now (List now for a full listing) and Recover, filtered by name, family, status or tier. A row expands to its details: the activity checkpoint and the windows of the last run for Audit activity, the facts written, the claims and the unresolved searches for Data usage, the resource windows and the capacity for Resource usage, the login and automation scans for Account signals, the datasets of a catalog, the backfill progress and its controls (Pause, Resume, Cancel backfill, Plan again). For Audit activity, Data usage, Resource usage, Account signals and Policies, Searches of the last run lists each search the run sent, exactly as sent, with its time range, Copy and Open in Search. Custom policies have their execution health in their own family.
Data & storage — what UAM writes for the tenant and how to read it: the events per index and sourcetype (volume over a range, latest event), the KV collections (records, size, lookup, and a record viewer), the logs and every scheduled report with its live state, each with a shortcut.
Troubleshoot & help — the diagnostic glossary (every code UAM can report: what it means, where it comes from, possible causes and what to do; the codes this tenant reports now come first) and ready-made searches (tracker errors, failed or incomplete runs, skipped scheduled reports, REST calls, who changed UAM).
Every diagnostic code, wherever it appears, opens its explanation. A code is coloured as a warning or a failure only while its condition is current; a code that only describes a past, recovered run stays neutral. Only one run of a report happens at a time; a run whose search provably stopped is recovered automatically after five minutes, otherwise Recover stopped run releases it. A lock is never taken over on age alone.
The Configuration Guardian check uam_collection_health warns when collection is not
running as configured — a missing or unscheduled report, missed schedules, repeated
failures, a checkpoint that does not move, a range held for 15 minutes, windows or recounts
waiting more than a day, an overdue listing, failing notification delivery, a custom policy
that drifts from its schedule. See Configuration Guardian.
AI Agent¶
The UAM AI Agent: whether scheduled triage is on, its totals over 24 hours to 90 days, Run a review now, and the decision log with its filters (decision, mode, outcome, scheduled or asked). See The UAM AI Agent and the AI Assistant.
Settings¶
Every tenant setting in one page with one Save settings: Environment (the tiers and the route account), Activity scope (advanced), Collection (scheduled trackers, Store SPL search, resource collection), Indexes, Collection limits (time budgets, windows and cadences: there is no row limit), Notifications to operators, Notifications to users, Account classification, Data usage (the VOL and DSM links, the excluded apps and the resolution limits), Privileged users, AI Agent, Assignment and Finding actions. Changes apply from the next runs; Collect now applies them immediately. If someone else saved in the meantime, your save is refused and you are asked to reload. Every setting is described in Settings reference.
Accounts and classification¶
A finding about an account only makes sense once you know what the account is. UAM keeps three classes, and the class decides which policies apply:
human — a person. Detected automatically from interactive activity: a search run from Splunk Web (the Search app, a dashboard, a report opened by a person) or a browser login to Splunk Web. One is enough, and the class stays.
service — automation. Detected after seven distinct days of automation-only activity (API jobs, the scheduler, acceleration) and nothing a person does; withdrawn as soon as the account does what a person does. This detection needs the classic
key=value_auditformat, and the seven days must be inside the audit retention.system — Splunk’s context identities (
nobody,splunk-system-user) and the built-inadmin, preset. System accounts are never evaluated by the account policies and never receive email.
On top of the detection, set manual classes per account (Users tab or Settings →
Account classification) and ordered classification rules (exact names or wildcard
patterns such as svc-*), with a preview of the effective class before saving. Precedence
is always: manual class, system defaults, the first matching rule, human detection, service
detection. An account nothing classifies stays unclassified and is left out of the
policies scoped to human accounts.
Tip
Classify the monitoring itself first. Classify the tenant owner (svc-a), your
administrator accounts and other platform automation as system or service: they
run TrackMe’s collection and the platform’s housekeeping and should not fill the
findings. The trackme and Splunk_SA_CIM apps are excluded from the policies by
default for the same reason.
Triage findings¶
Observed, not observed, and the status¶
A finding has two independent axes:
Whether the policy still sees the condition. While it does, the finding is observed. After a complete policies run that no longer sees it, the finding reads not observed — UAM never resolves it for you: a configuration or a behaviour that disappears from one evaluation still deserves a human decision. A run that could not cover everything (an incomplete inventory, an activity gap) may add findings, never clear them.
Your decision, the status: Open, Investigating, Resolved or Dismissed.
A resolved finding that is observed again is reopened. A dismissed finding stays dismissed until its dismissal ends — then it returns to open, so you get another look. The lifecycle shared with the other findings tenants is described in A finding and its lifecycle.
Finding groups¶
All the findings about one account and the saved searches it owns form one Finding
group: the all-time searches and resource consumption of analyst-a together with the
personal-schedule and real-time-schedule findings of the searches analyst-a owns. A
saved search without an owner keeps a group of its own. You decide once, for the group:
status, dismissal, assignee, notification snooze and severity apply to every member, and a
finding that joins the group later takes its state.
Two UAM specifics:
Dismiss only this finding (member menu) sets one finding aside — for example a personal schedule you accept inside a group that also holds a real problem. It stays dismissed on its own while the rest of the group is decided on; Reopen this finding brings it back into the group’s state.
A new kind of finding on a dismissed group returns the group to open with the new finding, while the findings the dismissal covered stay dismissed: a dismissal never silences a new problem.
You can also group by hand — Merge into one Finding group (2 to 25 groups), Add to the Finding group (an existing manual group), Remove from the Finding group, Ungroup — for example to handle the accounts of one project together. The findings of one account always move together. See Finding groups.
Decisions¶
From the row, group and member menus, the selection toolbar (Bulk edit) and the Triage panel of the Investigate view:
Decision |
What it does |
|---|---|
Mark investigating |
Someone is on it. |
Resolve |
The investigation concluded. The policy reopens the finding if it observes the condition again. Nothing is changed on the saved search or the account. |
Dismiss |
No action needed: for 1, 7, 30 or 90 days, or until reopened by hand (permanent). A dismissed finding stays listed under the Dismissed tile. |
Reopen |
Back to open; on a dismissed finding, ends the dismissal now. |
Assign to me, Assign to…, Unassign |
To a Splunk user or an assignment group. The assignee is kept when the finding reopens. |
Snooze notifications |
Holds the operator emails (from 1 hour to 30 days) without changing the status; Clear notification snooze resumes them. It does not hold the notifications to users. |
Set severity |
Overrides the policy severity (Low, Medium, High, Critical); the override survives later runs. Choosing the derived severity again (Members on a Finding group, Use policy severity on a single finding) restores it. |
Notes |
Markdown notes on a finding or a Finding group, permanent or expiring, plus a short investigation note recorded with each decision. |
Exclude owner from this policy |
Adds the finding’s owner to the exclusions of its built-in policy (custom policies write their exclusions in their SPL). |
The row menu also has Send email communication, Send webhook… and your own actions under Custom actions, Ask the AI Agent…, the account view and saved search view, Copy finding ID and Open record / events in search.
Every decision is recorded in the history of the finding and of its Finding group, with the actor, the time and the note. Decisions are protected against concurrent changes: if the Finding group changed since you opened it, your decision is refused and the table reloads.
Assignment and findings actions¶
Assignment groups — “Platform team”, “Security operations” — and ordered routing rules (by policy, severity, subject type, owner or account, app, environment) are set in Settings → Assignment; routing assigns new Finding groups automatically, never overriding an existing owner, and can email the assigned team. See Assignment and automatic routing.
Findings actions (Settings → Finding actions) add drilldown searches, links,
webhooks through system-wide Webhook accounts, and the email action to the finding
menus, by hand or through per-policy automation rules (policy editor → Automations).
UAM presets include Audit activity of this account, Scheduler runs of this saved search
and Finding transitions. Tokens such as $subject.account$, $subject.owner$,
$subject.app$ and $subject.name$ carry the finding’s values. See
Findings actions and Webhook accounts.
Every finding transition is also indexed in the tenant’s notable index as
sourcetype=trackme:uam:finding, ready for your own Splunk alerts, Enterprise Security
correlations or SOAR playbooks.
Communicating with users¶
UAM can talk to two audiences: the operators who triage, and the accounts the findings are about. The two are independent.
Notifications to operators¶
Settings → Notifications to operators (off by default): the events that send an email (New finding, Finding reopened, Severity raised, Finding resolved, and optionally Finding dismissed and Finding assigned), the minimum severity (Medium), the policies (all), the email delivery account, up to 50 recipients, and the mode: a Digest (one email per policies run) or Per finding (one thread per finding). Emails are sent at the end of the next policies run, within about 15 minutes, and link to the finding. Send test email uses the saved settings, even while notifications are off.
Notifications to users¶
Settings → Notifications to users (off by default) emails the account a finding is about, so that people fix their own searches:
on a New detection, and when the Condition cleared (after two consecutive complete evaluations without it);
addresses come from Splunk’s user records on each tier, or from an administrator’s Override email address; system accounts never receive mail, human accounts do by default, service and unclassified accounts only if you opt in;
a common banner, introduction, footer, reply-to address and HTTPS support link, combined with each policy’s own subject and body (Policies → Edit → Email template);
limits: 50 emails per run, 3 per address per hour, 24 hours between repeat detections.
When you turn it on, UAM records the current findings as a baseline and only emails what is detected from then on, unless you choose Include current detections once. User emails follow the policy’s detection, not your triage: resolving, dismissing or snoozing a finding does not stop them.
Important
Before turning on notifications to users, check the address coverage shown in the section, set the reply-to address and the support link to your team, and preview the email template of each enabled policy. The people who receive these emails did not ask for them: make the first one clear and useful.
Send email communication¶
For a one-off message, Send email communication in a finding, member or Finding group menu prepares an email to the account concerned — one finding, or a whole group (up to 200 findings) in one email. You review the preview, can change the recipient once, and confirm explicitly. The email is sent by a TrackMe search job, never by the browser, and a delivery that could not be confirmed is reported, never resent blindly.
From a Finding group menu, the same dialog prepares one email for every finding of the group:
here the three findings of jane.doe — a built-in activity policy, a custom policy and the
CSV lookup footprint — under the subject 3 user activity findings.
The UAM AI Agent and the AI Assistant¶
The UAM AI Agent reviews findings the way an operator would: it reads the evidence
TrackMe holds and writes its assessment as one current [AI Agent] note per finding.
Inspect changes nothing: assessment, confidence and recommendation.
Act also applies the decisions the tenant allows: mark investigating and dismiss for a time (on by default, never for a critical or high finding still observed), resolve a finding its policy no longer observes and classify an unclassified account (off by default). It never dismisses permanently, assigns, or changes a severity, a policy or a setting.
Ask the AI Agent… in a finding or group menu inspects one Finding group on demand; you
can then let it act on that inspection. Scheduled triage (Settings → AI Agent, off by
default) reviews up to 10 findings every hour — by default only critical and high ones,
one per Finding group — and never overrides a decision or an assignment made by a person.
Every decision appears under the actor AI Agent in the history and in the AI Agent
tab. The AI Agent needs the AI Assistant to be enabled with at least one AI provider; the
scheduled triage also needs a tenant owner other than nobody or splunk-system-user.
The UAM AI Agent replaces the UAM Findings Advisor of earlier releases. See
Findings AI Agents and AI Agents for findings.
Custom instructions. The tenant administrators can give the agent instructions in plain
English (Settings → AI Agent → Custom instructions, also in Manage AI Agents automation and
in the Configure tenant dialog): at most 4,000 characters, line breaks kept. Every run of
the agent follows them, asked or scheduled — for example, “The accounts named svc_* are
service accounts.” They never widen what the agent may do: the allowed decisions and
TrackMe’s checks before every decision still decide. The tenant’s shared Custom
Instructions of Manage AI Agents automation are different: they apply to scheduled runs
only.
Evaluate… tries the unsaved draft before you save it: the agent runs in inspect mode on
one open finding, or on a review of the open findings (at most 10, the most severe first),
optionally side by side with a run without custom instructions. An evaluation writes nothing
— no note, no decision, no review record — and is recorded with the status evaluation.
See AI Agents for findings.
The AI Assistant of a UAM tenant knows the workspace, the policies and the open findings: ask it to explain a finding, compare accounts or suggest what to tune, and it can propose an AI Agent review or a triage action for you to confirm. See AI Assistant & Concierge.
Tuning the policies¶
The built-in catalogue¶
Sixteen policies in four families. Seven are enabled on a new tenant; the four privileged policies are off until you turn them on.
Policy |
Family |
Default |
What it detects |
|---|---|---|---|
Scheduled search owned by a human account ( |
inventory |
on, Low |
An enabled schedule owned by a person: it stops when that account goes. |
Scheduled search owner is not classified ( |
inventory |
off, Medium |
An enabled schedule whose owner is neither human nor service. |
Scheduled search exceeds the hourly schedule limit ( |
inventory |
off, Medium |
A cron with more minute slots per active hour than the limit (12). |
Scheduled search has an all-time dispatch lower bound ( |
inventory |
on, Medium |
A schedule whose earliest time is |
Scheduled search uses a real-time dispatch range ( |
inventory |
on, Medium |
A real-time scheduled search. |
CSV lookups attributed to an account exceed its size allowance
( |
inventory |
on, Medium |
Per tier, one CSV lookup over the file limit (100 MiB) or an account’s CSV lookups over its total (500 MiB), attributed to the probable writer, else the recorded owner. |
Account launched an unusual number of searches ( |
activity |
off, Medium |
More distinct searches in the policy window than the limit (200). |
Account ran repeated long searches ( |
activity |
off, Medium |
More than 3 searches of at least 300 seconds. |
Account ran all-time searches ( |
activity |
on, High |
Any all-time search (the maximum is 0). |
Account’s search resource consumption exceeded its allowance
( |
resource |
on, Medium |
CPU over the allowance in the window (10 % of the search capacity by default); the severity is raised one level when the account dominates the environment (40 %). |
Account ran repeated resource-intensive searches ( |
resource |
on, Medium |
More than 3 searches heavy in CPU or memory. |
Account’s CPU consumption is unusual for its daily baseline
( |
resource |
off, Medium |
Today’s CPU above a multiple (3) of the account’s daily 95th percentile over 14 days. Ships in Preview mode: it shows what it would match, without raising findings or sending notifications, until you turn the mode off. |
Account became privileged ( |
privileged |
off, High |
An account that became privileged: created privileged, given a privileged role, or a role it holds now imports one. Reported once per change, with who made it when Splunk audited it. |
Role gained a privileged capability ( |
privileged |
off, High |
A role whose own capabilities or imported roles changed so that it grants more privileges, or a new role created privileged. One finding about the role, listing the roles that inherit it and the accounts newly privileged through them. |
Privileged account is dormant ( |
privileged |
off, Medium |
A privileged account with no search, no Splunk Web login and no successful login reported by Splunk for 30 days. |
Privileged accounts differ from the reference ( |
privileged |
off, High |
A privileged account your reference (typed names and lookup tables) does not declare; optionally a declared account that holds no privilege or no longer exists (Low). |
The activity and resource policies judge 15-minute windows of collected evidence by default. A window ends where the collected evidence ends, never at the current time, and consecutive windows always touch: no minute is skipped, and activity collected late is evaluated once it is collected. The inventory policies follow the catalog cadence (the full saved-search listing is daily by default). The resource policies stay learning — no finding, nothing cleared — until the tenant holds 24 hours of resource history and a measured capacity. The full catalogue, with every threshold and its bounds, is in The built-in policy catalogue.
Editing a policy¶
Edit opens the policy editor:
General settings — status, During bank holidays (see Bank holidays), severity, account scope (human accounts only, classified accounts, all accounts), thresholds (the resource policies can derive them from the measured capacity, auto thresholds), excluded owners and excluded apps (exact names, up to 500), and Preview policy to see what the draft would raise against the current data before you save.
How it is evaluated — the rule in plain words, then the exact searches it runs (built from the saved settings, with their time range, Copy and Open in Search), or, for an inventory policy, the records it judges.
Email template — the subject and body of the user emails for this policy, for a new detection and for a cleared condition, with a preview and a test delivery; Restore policy default brings back the shipped text.
Actions and Automations — the findings actions offered on this policy’s findings, and the automation rules that run a webhook when they change.
The policy insight (from the last evaluation of a policy) explains a run: the verdict, what the run saw, what was matched, excluded, not applicable or could not be judged and why, the owners behind the unknowns (classifiable in place), the exclusions and a seven-day history, with the runs of those seven days that could not judge everything and their reasons (kept after the problem recovered). It is the first place to look when a policy raises nothing, or too much.
Privileged policies¶
Four policies watch who holds high privileges, from the assessment of the Privileged users tab (its rules: Settings → Privileged users). They are off by default: turn on the ones you need in Policies.
Account became privileged reports, once, an account that became privileged on a tier: a new account created privileged, a privileged role assigned to it, or a role it holds that now imports a privileged role. When Splunk audited the change, the finding names who made it and when; a role mapped by a SAML or LDAP provider has no such record.
Role gained a privileged capability reports, once, a role whose own definition changed so that it grants more: its own capabilities, the roles it imports, or a new role created privileged. Inheritance is transitive, so one change reaches every role that inherits it, directly or through other roles: the finding is about the role whose definition changed, and lists the roles that inherit it and the accounts newly privileged through them — one change, one finding.
Privileged account is dormant reports a privileged account with no activity for 30 days (configurable): no search, no Splunk Web login, no successful login that Splunk reports on any tier. It judges an account only once it has been in the inventory and the tenant’s evidence covers the whole period.
Privileged accounts differ from the reference compares the privileged accounts with a reference you maintain — the accounts declared privileged, optionally per search head tier. It reports a privileged account the reference does not declare (High) and, if you keep Report stale references on, a declared account that holds no privilege or no longer exists (Low).
The reference is set in the policy editor, from two sources that add up, case-insensitive:
Account names typed in the policy (up to 1,000), declared on every search head tier;
*matches any text (svc-admin-*).Lookup tables (up to 20): Add a lookup table picks the Splunk deployment (the local search tier, or a remote deployment through a TrackMe remote account), the lookup (a CSV or KV store lookup shared with the TrackMe app, with a preview of its first rows), the account field, an optional tier field (the search head tier a row declares the account for:
localor a remote account name; not set: every tier) and the match mode — Exact (a value holding*is ignored) or Wildcard. Each lookup holds up to 500,000 values and is read at every policies run; a lookup that cannot be read, or reads empty, keeps its last good values, and one never read yet makes the policy judge nothing until it can be read. TrackMe’s own lookups are not offered.
Test reference reads every lookup now, as a search, and shows what each source declares, the privileged accounts the policy would report as undeclared, the stale references and the patterns that declare nothing. It saves nothing.
The first evaluation of a change policy records the current state and reports nothing, and a
change of the Privileged users rules records the new state without reporting. The accounts
Splunk operates on a Splunk Cloud stack (their email address is support@splunk.com) are
left out unless you include them. These policies never email the account a finding is about;
notifications to operators work as for every policy. A role finding has no account to write to:
Send email communication is not offered on it.
Custom policies¶
When the catalogue does not cover a rule of your own, write it in SPL: Policies → Create policy (TrackMe administrators of the tenant).
Definition — title, description, default severity and remediation.
Search and schedule — the search source (an account policy reads through the route account), the app context, the SPL (up to 64 KiB), a relative window ending now (at most 31 days) and a five-field cron.
Map search results to findings — the subject type, Account or Saved search (owner, app, name), and which result fields carry the identity, dimensions, evidence, count and time.
Preview policy — runs the draft as you and shows up to 20 candidate findings. A preview with no results cannot verify the mapping.
Save draft, Publish (an immutable revision), Enable (a scheduled search for the published revision). Version history lets you load, compare, restore or delete versions; Disable, Archive and Delete permanently end a policy (its findings, evidence and notes remain).
The definition also has a Bank holidays section (see Bank holidays). Custom findings appear with the built-in ones, join the Finding groups and are triaged the same way. In this release, operator and user notifications cover the built-in policies only, and a custom policy writes its exclusions in its SPL. A tenant holds up to 100 custom policies, 25 of them enabled; their execution health is in Collection health → Trackers, family Custom policies.
Bank holidays¶
A UAM tenant uses the bank holiday calendars like every tenant: the system default calendars, unless it selects its own or none (Manage Bank Holidays → Tenants & settings, or Configure tenant → Bank Holidays). One effect applies to it, Suppress new alerts, shown as Hold new findings and on by default: while a holiday of the tenant’s calendars is active, the policies set to hold open no new finding. SLA, monitoring windows and ML training do not apply to a UAM tenant.
Each policy chooses in its editor, under During bank holidays: Hold new findings or Keep opening findings. The defaults:
Holds new findings |
Keeps opening findings |
|---|---|
Scheduled search owned by a human account, Scheduled search owner is not classified, Scheduled search exceeds the hourly schedule limit, Scheduled search has an all-time dispatch lower bound, Scheduled search uses a real-time dispatch range, CSV lookups attributed to an account exceed its size allowance, Privileged account is dormant |
Account launched an unusual number of searches, Account ran repeated long searches, Account ran all-time searches, Account’s search resource consumption exceeded its allowance, Account ran repeated resource-intensive searches, Account’s CPU consumption is unusual for its daily baseline, Account became privileged, Role gained a privileged capability, Privileged accounts differ from the reference |
The inventory policies and the dormancy policy are reviews: nobody reviews them on a day off, and they are still there the next working day. A privilege granted on a day off is more telling, not less. The activity and resource policies keep opening findings, because unusual activity on a day off is more telling, not less. A configuration saved before TrackMe 2.4.19 reads each policy’s default.
When a policy holds:
An inventory policy holds while a holiday is active at the time of the run; an activity or resource policy holds when a holiday overlaps its activity window, so a burst on the last evening of a holiday is still a holiday burst.
A held match writes nothing: no finding, no Finding group, no notification to operators or users, no findings action, and it uses none of the findings capacity. The first policies run after the holiday opens the finding if the condition is still there.
A finding already observed is evaluated as on any other day: updated, escalated, or no longer observed. A held match does not reopen a resolved or dismissed finding.
If the calendars cannot be read, nothing is held.
A custom policy keeps opening findings unless its definition turns the setting on (Bank holidays section of the custom policy editor; it takes effect when published). It holds while a holiday is active at the end of its search window.
Where to see it. Each policy on the Policies tab says During bank holidays: holds new findings or keeps opening findings. A policy that held matches shows held N (bank holiday) under its last evaluation, and its policy insight says Held N (bank holiday: <names>) with a Bank holidays line for the run. The policies run receipt records the state (holding, off or unavailable), the holiday names, the matches held and the enabled policies set to hold; a custom policy run records what it held the same way.
Upgrading to TrackMe 2.4.21¶
Three trackers for the indexing layer. What the activity tracker did in one run is now Audit activity, Resource usage and Account signals, each with its own schedule, time budget, lock and health. The two new reports are created automatically within five minutes of the upgrade, with the tenant’s owner; they continue where the earlier tracker stopped, and nothing is collected twice.
No row limits any more. The seven record-count settings of Collection limits are removed: what a run cannot finish in its time continues at the next run. Stored values are ignored and disappear at the next save; an API client that still sends them keeps working.
New time budgets. Every run budget defaults to its tracker’s interval less 30 seconds. A budget still at its earlier default moves to the new one; a value you chose is kept. A scheduled run dispatched late ends 30 seconds before the next scheduled run, so the scheduler never skips one.
Correlated executions are no longer indexed. They are correlated from the audit records when they are read. Your own searches, dashboards, alerts or custom policies on
sourcetype=trackme:uam:executionreturn nothing for activity after the upgrade: move them tosourcetype=trackme:uam:observation. The UAM events also carry fewer fields (see The evidence).Backfill. A new tenant starts without backfill unless you choose a range, in 5-minute buckets. An existing tenant keeps its bucket size; set Backfill bucket (s) to 300 to move it to 5-minute buckets (the coverage already collected is kept).
Splunk restart. Some index-time settings of the UAM events take effect after the splunkd restart of the upgrade, on the instances that parse the UAM log.
Privileged users and privileged policies. The Privileged users tab and its settings appear with the release defaults (
*admin*,trackme_adminexcluded). The four privileged policies are off; when you turn on a change policy, its first evaluation records the current state and reports nothing. The users catalog now also collects the last successful login Splunk reports (never an inventory change).
Upgrading from TrackMe 2.4.18¶
The first policies runs gather the existing findings into Finding groups; decisions, notes and history are kept.
The inventory moves to the REST API catalogs: each tier gets its new knowledge-object reports, and its first full listing runs within the hour. The users and roles reports move from a 5-minute to a 15-minute schedule.
The UAM Findings Advisor is replaced by the UAM AI Agent; an enabled advisor’s settings are carried over once.
Where to go next¶
See also
User Activity Monitoring — reference — the reference: tiers, trackers and schedules, evidence, classification, the full policy catalogue (The built-in policy catalogue), every setting (Settings reference), REST and troubleshooting (Troubleshooting and known limits).
Findings, Finding groups and findings actions — findings, Finding groups, assignment, notifications, findings actions, Webhook accounts and the findings AI Agents, shared with the other dedicated tenant types.
Creating a splk-uam tenant — the tenant creation wizard.
Who is doing what on your Splunk? User Activity Monitoring (UAM) — the white paper.
User Activity Monitoring — the hands-on deck.
Configuration Guardian — the
uam_collection_healthcheck.The AI Advisors and AI Agents automation — the AI Agents and their automation.