User Activity Monitoring

TrackMe has always watched the data. User Activity Monitoring watches the other half of a Splunk platform: the people and the automation that use it. It answers a question none of the entity components asks — who is doing what on my Splunk? Which accounts run searches, how often and at what cost; which scheduled searches are owned by a person who may leave tomorrow; which account started to run all-time searches; who consumes the search capacity of the platform; whose CSV lookups keep growing; which indexes are read, by whom, and which receive data no one reads.

A User Activity Monitoring tenant (UAM) reads what Splunk already keeps — the audit trail, the search resource records and the knowledge objects of every search head — and turns it into findings: one finding per policy and subject, with its evidence, a severity, an investigation status, an assignee, notes and a history. You investigate, decide, notify the people concerned and move on.

UAM workspace, Findings tab, an account Finding group expanded (jane.doe) with a CSV lookup footprint finding among its members, tenant uam-example

UAM is a dedicated tenant type, not an eighth entity component: a tenant is either an entity tenant (DSM, DHM, MHM, FLX, FQM, WLK, VOL) or a UAM tenant, never both. There is no decision maker, no impact score and no state colour behind it: what you work with are findings. UAM shares its findings model — Finding groups, triage, assignment, notifications, findings actions, the AI Agent — with the other dedicated tenant types (Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream and Cribl Edge); those concepts are explained once in Findings, Finding groups and findings actions, and this page documents how they work in UAM.

Tip

This page is the task-oriented guide: set the tenant up, find your way in the workspace, triage, communicate and tune. Every setting with its default and bounds, the schedules, the indexed evidence, the REST API and the troubleshooting table are in User Activity Monitoring — reference.

What UAM watches

One UAM tenant monitors one indexing layer seen through one to ten search head tiers: the local search head and / or remote Splunk deployments reached through TrackMe remote accounts, on Splunk Enterprise or Splunk Cloud.

What

What UAM knows about it

Accounts

Every account seen in the users inventory or in the search activity — people, service accounts and Splunk’s own system identities — with its class (human, service or system), its roles, the tiers it exists on and its email address when Splunk has one.

Search activity

One compact audit record per search and collection window, read from _audit: the account Splunk reports, the app, the search type, the outcome, the run time, and whether the search covered all time. Executions are correlated per search.

Resource usage

The CPU time and peak memory of every search, on the search heads and on the indexers, and each account’s share of the environment’s search capacity. CPU follows Splunk’s own accounting where it exists.

Saved searches

The saved-search inventory of every tier: owner, app, schedule, dispatch time range, enablement, and every change (added, changed, removed).

Other knowledge objects

Dashboards, macros, event types, field extractions and transforms, field aliases and calculated fields, lookup definitions, automatic lookups, CSV lookup files (size, owner, modification time — never their contents) and tags, read from the Splunk REST API of each tier.

Data usage

The indexes every search names, by account, app and path (literal term, macro, event type, saved search, data model or wildcard), the knowledge objects that declare each index, and — through linked VOL and DSM tenants — what each index ingests and costs. See Data usage.

Findings

What the policies raise: sixteen built-in policies and the custom policies you write in SPL.

Note

What UAM claims, and what it does not.

  • An account in UAM is the account reported by Splunk: it is not a verified person. Shared credentials or an integration running as a person’s account look like that person.

  • Collection is bounded by time only, and nothing is dropped to fit a row count: what a run cannot finish in its time continues at the next run. Coverage still depends on what the Splunk sources retain and what the collection accounts can see; gaps and partial coverage are recorded and shown in Collection health, never hidden.

  • There is no machine learning in UAM: thresholds are explicit, and the daily CPU baseline is a percentile over the account’s recent days.

  • UAM never changes, dispatches or deletes anything it observes. A decision on a finding stays in TrackMe.

Before you start

A licence beyond Foundation. UAM is available with the Enterprise and Unlimited editions, the Unlimited trial and the Developer licence. It is not available on the Foundation edition or the Foundation trial. When a licence expires, the workspace stays readable while changes are refused, and scheduled collection continues until the licence enforcement ladder stops the tenants.

The Splunk sources. UAM reads _audit (search activity), _introspection (search resource records and capacity) and the indexers’ search completion records (CPU accounting), through one search head tier called the route account. Splunk Enterprise 9.4, 10.4 and 10.6 and Splunk Cloud 10.5 are qualified; other versions run as well and only add a soft diagnostic in Collection health. Three things to know:

  • The automatic detection of service accounts relies on the classic key=value format of the _audit lines. A deployment whose _audit only contains the JSON format gets no automatic service detection (human detection and your own rules still work).

  • What UAM can look back on is bounded by the retention of these indexes. _introspection is often kept for a shorter time than _audit, so resource history may end earlier than audit history.

  • Before Splunk 10.4, the lookup file listing gives no file size: UAM reads the sizes of the CSV lookup files through Splunk’s file explorer endpoint (admin/file-explorer), so the collection account of each tier must be allowed to read it. By default only the admin and splunk-system-role roles can, so a least-privilege account cannot retrieve CSV lookup sizes before Splunk 10.4. When the read is refused, Collection health reports Lookup sizes refused and the CSV lookup footprint policy cannot judge that tier. From Splunk 10.4, the listing reports the size and no extra access is needed.

Remote tiers. Each additional search head tier is a TrackMe remote account (Configuration → Remote accounts, see Remote Splunk deployments). The wizard tests each one. Live reads you make from the workspace as yourself — Fetch SPL, Fetch definition — also need your roles to be allowed on that remote account.

A tenant owner. Scheduled collection runs as the tenant owner. Prefer a dedicated service account (svc-a in the examples of this page) over nobody: the scheduled UAM AI Agent cannot run as nobody or splunk-system-user.

The collection accounts and their capabilities. Each search head tier is collected as one account: the tenant owner on the local search head, the token account of its remote account on a remote tier. UAM needs no list_settings and no list_storage_passwords. TrackMe checks these accounts when the tenant is created and on every health cycle. A gap raises the Configuration Guardian alert UAM collection permissions need attention, which names the tier and what is missing (Configuration Guardian).

Capability

Local tier (tenant owner)

Remote tier (token account)

What it is for

search

Required

Required

Run the collection searches.

trackmeadminoperations (or admin_all_objects)

Required, with one of the tenant’s admin roles

Required

The UAM administrative boundary. On the local tier, users and roles are read through TrackMe’s identity reader, which serves the tenant’s administrators only. The capability is defined by the TrackMe app: on a remote deployment where TrackMe is not installed, only admin_all_objects satisfies it.

list_all_users and list_all_roles (edit_user and edit_roles also suffice)

Not required

Required

List every account and role of the tier. Without them, Splunk lists only the token account and its own roles, and the tier’s users and roles catalogs hold one account. On the local tier, TrackMe reads them with its system token for the tenant owner.

list_search_head_clustering

On a search head cluster

On a search head cluster

Read the cluster members, to attribute activity to the tier.

Beyond capabilities:

  • The route account must be allowed to search _audit, _introspection and _internal (the indexers’ search completion records, and the searches they received, which Data usage reads).

  • Before Splunk 10.4, the collection account of each tier reads admin/file-explorer for the lookup sizes (see above).

  • Index permissions, knowledge-object permissions and the search scope of each account still decide what UAM sees. The check confirms the capabilities, not complete visibility: Collection health reports Users or roles listed as one account sees them (identity_visibility_limited) when a tier’s listing still holds only the reading account.

Configuration Guardian alert UAM collection permissions need attention, warning, on the UAM tenant: the remediation in four steps — inspect the affected tiers (local checks use the tenant owner, remote checks the token account), grant only the reported requirements (list_all_users / list_all_roles, or edit_user / edit_roles, for a remote inventory; list_search_head_clustering for a search head cluster), run the check again and collect the catalogs, and the reminder that index and object permissions still decide coverage

Who can use it. UAM evidence is audit data, so the workspace is for the TrackMe administrators of the tenant — accounts holding the TrackMe administrator capability and one of the tenant’s admin roles; Splunk administrators can do everything on every tenant. Power users of the tenant have read access through the REST API (and can send user communications and run findings actions through it); read-only users have no UAM access. The workspace’s own searches (the Data usage views, the Privileged users activity, Fetch SPL) run as you, so your roles also need the tenant’s TrackMe indexes, and the search text index when it is a restricted one.

Email delivery (optional). Notifications and communications are sent through a TrackMe email delivery account (Configuration → Emails), or the local mail transfer agent when none is selected. The account’s allowed-domain rules apply.

Note

UAM indexes its evidence in the tenant’s summary index and its finding events in the tenant’s notable index, as trackme:uam:* events. This evidence counts against your Splunk licence like any TrackMe summary data; it is compact by design (one record per search and window). Some indexed fields of these events take effect after the splunkd restart that follows the app installation or upgrade; until then, reads fall back to slower searches.

Create the tenant

From the Virtual Tenants page, open Actions → Create a tracking tenant and choose the User Activity Monitoring card (Investigate Splunk user activity and policy findings). The wizard has four steps.

The tenant creation chooser: User Activity Monitoring is the first dedicated workspace, after the Splunk data monitoring components, followed by Cribl Stream, Cribl Edge, Splunk Forwarder Collectors, Splunk HEC, Splunk S2S and Splunk UF Fleet

Step 1 — Basics

The tenant identifier (lowercase letters, digits and hyphens, up to 20 characters; it cannot change later), an optional alias shown on the tenant card and an optional description.

Step 2 — Configure UAM

Search head tiers. Local search head is selected by default; each TrackMe remote account can be added as another tier, up to ten. Select every search head tier that shares the indexing layer you want to monitor — for example an ad hoc search head cluster and an Enterprise Security cluster on the same indexers. Remote accounts are tested as soon as you select them, and creation waits until every selected one is reachable (Retry connection test if one fails).

Route account. Derived from your selection: the local search head when it is selected, otherwise the first remote tier. The indexing-layer searches — audit activity, resource usage, capacity, logins and automation — go through it, once for the whole layer, while the knowledge objects are read on every tier.

One indexing layer per tenant. All the tiers of a tenant must search the same indexers. Monitoring two independent deployments takes two tenants.

Historical backfill. How far back UAM collects the audit and resource evidence that existed before the tenant: No backfill (the default), 7, 14, 30, 60 or 90 days. The backfill runs hourly in 5-minute buckets, newest history first, and retires itself once the range is covered or the retention of the sources is reached. It makes the account classification and the activity history useful from the first day, at the cost of reading that history from the indexers: on a busy deployment, start without it, or with a short range, and extend it later in Settings → Collection limits.

Activity scope (advanced). Optional hostname patterns (* and ?, up to 20 include and 20 exclude patterns) that restrict which search heads’ activity is collected, for example to leave out a test search head that shares the indexers. Exclusions win. The scope is part of the base searches, so the indexers only return the events of the search heads in scope: on a large indexing layer shared by several search head clusters, it is also the first way to make collection lighter.

Wizard step Configure UAM with two search head tiers (local search head and a remote account, the route marked), the route account line and Historical backfill set to 30 days

Step 3 — Indexes & RBAC

The TrackMe indexes of the tenant (summary, audit, metric and notable), the tenant owner and the admin, power and user roles. The summary index receives the evidence and the notable index the finding events; each family can be pointed at another index later in Settings → Indexes.

Step 4 — Review

Check the summary — the tiers with the route, the owner, the activity scope, the backfill — and create. Everything is provisioned and scheduled at once: the inventory, audit activity and resource usage reports start within minutes, the account signals and the policies follow, and each tier’s first full saved-search listing runs within the hour. If the creation result is incomplete, the wizard says so and the workspace offers Repair UAM workspace.

The step-by-step creation is also described in Creating a splk-uam tenant.

After creation

The tenant appears on the Virtual Tenants page with its open Finding groups and findings, the high or critical ones, and its collection status — here the historical backfill in progress. Selecting it opens the workspace; double-clicking it opens a read-only overview.

Virtual Tenants page, the UAM tenant card with its open Finding groups and findings, one high or critical, and the backfill progress

Your first hour with a new tenant:

  1. Open Collection health: the Status view says whether anything needs attention, and Trackers shows the first inventory, audit activity, resource usage, account signals and policies runs of every tier.

  2. Classify the monitoring accounts as system — the tenant owner, your administrator accounts — so that they never appear in findings (see Accounts and classification).

  3. Review the Policies tab: the enabled policies, their severities and exclusions.

  4. Come back to Findings after the next policies run (every 15 minutes).

The workspace

The workspace opens on the Findings tab. At the top, the status strip tells you that the tenant is working: the search head tiers (the route account marked · route), the age of the last Inventory, Audit activity, Resource usage, Account signals, Data usage and Policies runs, the activity checkpoint lag and coverage, the backfill progress, a Configuration changed since last run warning after a settings change, Refresh, and Collect now — which runs every tier’s inventory, then the audit activity, the data usage, the resource usage and the account signals, then the policies, with the current configuration, and every tier’s full saved-search listing alongside.

UAM status strip with the tier pill (marked route), the Inventory, Audit activity, Resource usage, Account signals and Policies chips with their age, the checkpoint lag and the coverage, Refresh and Collect now

Eleven tabs follow: Findings, Users, Privileged users, Data usage, Activity, Scheduled searches, Policies, Audit changes, Collection health, AI Agent and Settings.

A few conventions apply everywhere, shared with the other findings workspaces (The shared workspace conventions):

  • Column headers filter and sort the loaded page; the toolbar filters search everything.

  • Every panel has a ⋮ menu with Open in Search (the search behind the figure, ready to edit) and Refresh.

  • Three personal display preferences per tenant live in the Tenant Home settings panel, under Personal preferences for this tenant: Collection monitors (the status strip), Single stats and Charts. They change your view only.

  • Exclude TrackMe activity (Users, Activity and the account view) leaves out the searches run in the trackme app — TrackMe’s own scheduled trackers (recognised by their search id) and your own use of TrackMe. An ad-hoc search that reports another app or none, such as the tracker searches TrackMe dispatches over REST, cannot be attributed and stays in. Findings are not affected. It is remembered per viewer.

  • Notification emails link straight to a finding or a Finding group, opened in its investigation.

Findings

Your investigation backlog. Tiles count Open groups, High / critical groups, Groups assigned to me, Resolved groups and Dismissed groups — Finding groups — plus New findings in range (findings first seen in the time range), and drive the filters; charts show the finding transitions over the time range (24 hours by default) and the open findings by severity and by policy. Filters cover the status, severity, policy, subject (accounts, saved searches or roles), first seen, assignment (including assignment groups) and free text (owner, app, search name).

The table lists one row per Finding group: an account and the saved searches it owns. A group of several findings has an expander showing each member with its own policy, severity and whether the latest policies run still observed it. Selecting rows offers Merge into one Finding group, Add to the Finding group (a manual group), Run action, Assign to… and Bulk edit; AI Agent review asks the AI Agent to review the open and investigating findings.

Findings tab: the tiles counted in Finding groups (9 open, 7 high or critical), the finding transitions of the last 24 hours, the findings by severity and by policy, and the filters

Selecting a row opens the Investigate view: the subject, the findings of the Finding group, What happened in plain words, the suggested remediation, the evidence of the selected finding, Triage of the Finding group (status, dismissal, assignee, notification snooze, severity, investigation note), the notes and both histories — the finding’s and the group’s. From there, open the account view or the saved search view, or the record and its events in Search.

Each finding explains its own evidence. A CSV lookup footprint finding, for instance, lists the files attributed to the account, their size and how each was attributed: to its probable writer — the account whose searches wrote the file — or, when no writer could be identified, to its recorded owner.

Users and the account view

Every account across the tiers: tiles for the accounts in range, inventoried, classified and unclassified; filters by class, source and activity range; columns for email addresses, class (with a detected marker when a detection set it), roles, tiers, findings, audit records, executions, last seen and apps. The row menu sets the class or overrides the email address used for user notifications (Override email address…); a selection can be classified at once, or decided on as a whole (resolve, dismiss or mark investigating its open findings).

Users tab with the tiles, the class chips with the detected marker, roles, tiers, findings, audit records, executions and last seen

Selecting an account opens the account view:

  • Activity & usage — executions and run time over time by search type, CPU over time, the daily CPU baseline, the resource usage (CPU time, peak memory, share of the environment), the most expensive and the most recent searches with Fetch SPL, the account’s findings and the scheduled searches it owns.

  • Data accesses — which indexes the account read, over time: a chart of its named reads per index and one lane per index with a marker for each observation (see Data usage).

  • The account’s catalogs, one tab per family: Saved searches & alerts, Dashboards, Macros, Event types, Data models, Indexes, Fields, Lookups and Tags — what this account owns on each tier.

The resource usage tells what an account costs: here a service account whose scheduled detection searches use 0.2% of the environment’s search CPU over the day, with the most expensive runs listed first. Searches the resource records could not measure are counted and shown as such, never estimated.

Account view of alice.moreau, the Saved searches & alerts catalog tab: the scheduled searches the account owns, one expanded with its schedule and search definition

Fetch SPL shows the text of a search. By default UAM does not store search text: it reads it live from _audit, as you, and stores nothing. If you switch on Store SPL search (Settings → Collection, off by default), the text is indexed in its own record in the search-text index — point that index at a restricted index first, since search text can contain literals and secrets. When a text cannot be shown, the reason is given.

Search knowledge objects (on the Users tab) searches the collected catalogs across every owner: a name or a wildcard pattern, filtered by object type, tier, app, owner and sharing. Use it to answer “who owns the macro app_a_base?” or “which dashboards use this name on which tier?”.

Search knowledge objects with a wildcard name pattern, results across owners and tiers with type, app, owner, sharing and size

Note

The catalogs reflect what the collection account can see on each tier: the coverage of private objects depends on its permissions. They are collected on a cadence (daily in the off-peak window by default for most families, every 15 minutes for CSV lookup files, hourly for users and roles), not as atomic snapshots. A listing too long for one run continues at the next one, and the previous catalog stays in use until the new one is complete.

Privileged users

Which accounts hold high privileges on the monitored search head tiers, and why. An account is privileged when one of its roles — or a role that role imports, however indirectly — is a privileged role or grants a privileged capability:

  • Privileged roles: every role whose name matches a pattern (*admin* by default, case-insensitive, * and ?), plus the roles you add by name, minus the roles you exclude. The patterns are live: a role created later is privileged as soon as its name matches. trackme_admin is excluded by default: TrackMe’s own administrator role holds TrackMe capabilities, not Splunk-level ones.

  • Privileged capabilities: admin_all_objects, change_authentication, delete_by_keyword, edit_roles, edit_tokens_all, edit_user, list_settings, list_storage_passwords and rest_apps_management by default, granted directly or through an imported role.

  • An exclusion only removes a role’s name rule: an excluded role that grants a privileged capability still makes its users privileged, through the capability.

The tab shows the privileged accounts and their share of all accounts, the privileged roles, the privileged capabilities in use and the privileged accounts active in the last 24 hours; charts of why accounts are privileged (role, capability or both), of their classes and of the accounts holding each privileged capability; the privileged accounts with their reasons in words (team_lead imports admin (pattern *admin*), secops grants list_storage_passwords), last seen and executions; and the roles of every tier, the privileged ones by default. An account present on several tiers is one account, each reason naming the tier that grants it.

Everything is read from the users and roles catalogs UAM already collects: the view follows their cadence (hourly by default) and makes no call to Splunk. The rules are set in Settings → Privileged users: the patterns, and two dual lists — the roles and the capabilities the tiers hold — where moving a role a pattern matches out of the privileged list excludes it, and moving any other role in adds it by name.

Splunk grants roles and capabilities through direct membership and through inheritance, and a role inherits from roles that inherit further: every reason keeps its chain — team_lead imports admin, team_lead inherits admin through ops, leaf inherits list_settings from base through mid. The full rules are in Privileged users.

Four privileged policies build on this assessment, all off by default: an account that became privileged, a role that gained a privileged capability, a dormant privileged account, and privileged accounts that differ from a reference you maintain. See Privileged policies.

Privileged users tab: 9 privileged accounts of 7025, 17 privileged roles, 8 privileged capabilities in use, 6 active in the last 24 hours; charts of why accounts are privileged, by class and the capabilities held; the privileged accounts with their tiers, class, roles, capabilities, reasons, last seen and executions

Data usage

Which indexes are read, by whom, and which receive data no one reads? UAM already reads every search from the audit trail. The Data usage tab turns those searches into a map of your data: for every index, the searches that named it, the accounts and apps behind them, the path they took (a literal index= term, a macro, an event type, a saved search, a data model or a wildcard), the knowledge objects that declare it, and — when the tenant is linked to a VOL or DSM tenant — what the index ingests per day and what it costs.

It rests on one rule: an absence of reads is a claim, made only when the evidence supports it. An index is never called unread from a few hours of collection, an incomplete read or a wildcard that may reach it. Every view says what it covers, in one coverage card under its controls: when the collection started and what that means for the view (“Collected since 27 Oct 2026 (11.8 % of the selected period). “Read by no one” is provisional until 3 Nov 2026.”), a fault with its cause, or Coverage complete.

Five views share the same controls: the search head tier, internal indexes (left out by default), the scope (TrackMe’s own searches and the excluded apps, set in Settings → Data usage) and the time range (the last 7 days by default).

Every tile, chart and table of the Overview, Value & cost, Knowledge objects and Evidence views has a three-dots menu with Open in Search. It opens the search over the recorded facts behind that view, with a note on what only the map adds when you read the view: the catalog expansion of wildcards, the role scopes and the VOL volumes. How each kind of claim is recorded is described in Data usage reference: how each claim is processed.

Overview — where the data goes. One sentence answers the question: Of 6.60 GB/day ingested, 1.73 GB/day is read by name, 1.52 GB/day only through wildcards and 0.00 GB/day by no one so far. A bar splits the volume by usage, and the hero shows it either as a flow — from data families (indexes sharing a name prefix) to their usage, then to the accounts that read them — or as a treemap of ingestion, one tile per index sized by its daily volume and coloured by its usage.

Data usage Overview, Where the data goes as a flow: of 6.60 GB/day ingested, 1.73 GB/day is read by name, 1.52 GB/day only through wildcards and 0.00 GB/day by no one so far; data families on the left (webserver, cribl_*, siem-*, trackme_*), their usage in the middle (Read by name, Wildcard only, Excluded apps only, TrackMe's own) and the readers on the right Data usage Overview with its controls (search head tier, internal indexes, scope, last 7 days), the coverage card, the usage sentence and bar, and the treemap of ingestion by usage: webserver read by name, cribl_metrics and os-unix-nmon-metrics wildcard only, trackme_summary TrackMe's own

The usage classes are decided across the whole indexing layer, never per tier: Read by name (a search named the index), Wildcard only (only wildcards such as index=cribl_* reach it: they do not prove which index supplied the events), No reader, Excluded apps only (only the apps you exclude read it, for example trackme) and TrackMe’s own (the indexes TrackMe writes to, never a review candidate). An index without a linked volume is counted but not weighed.

Under the hero, three panels list the index references by path, the index usage verdicts and the reference paths (literal, macro, event type, saved search, data model).

Overview panels: index references by path (cim_modactions 488, webserver 163, license_generator 75, ...) with the literal, macro and data model split, the index usage donut (Searched 4, Indirect only 119, No observed reads 0) and the reference paths (literal 2584, macro 60, datamodel 38)

Value & cost — weigh what an index costs against how it is read. The treemap of ingestion by usage sits beside Volume against reads (log scales, one dot per index: the shaded corner holds the indexes larger than the median and read fewer than ten times), and Cost of disuse states what the indexes no one accessed, and those only wildcards reach, cost per day.

Value & cost: the coverage card, Ingestion by usage as a treemap, Volume against reads on log scales with the larger-than-median, read-fewer-than-10-times corner shaded, and Cost of disuse: 0.00 GB/day to indexes no one accessed so far and 1.52 GB/day, 1.29 USD/day, to 72 only wildcards reach

The Decisions list turns it into work: one row per index, largest first, with its daily volume and cost, its usage, who read it, its reads per GB, the knowledge objects that declare it and a suggested step (“Only wildcard searches reach it: confirm those searches need this data, or scope them by name”). Five views filter it: To review (no reader, wildcard only, excluded apps only and the least accessed), No reader, Wildcard only, Least accessed (the quarter of the indexes read by name that are read the least for their volume; the ranking needs at least four such indexes with a daily volume) and Every index. An empty list says why — for example, while the collection is younger than the period, nothing is called unread. The list pages 10, 25, 50 or 100 rows.

Decisions list, To review view, 116 indexes largest first: cribl_metrics, os-unix-nmon-metrics and em_metrics wildcard only, summary and siem-wineventlog-* excluded apps only, with GB/day, cost per day, usage, read by, reads per GB, declared by and the suggested step

Index activity — investigate an index. One row per index and tier: the incoming data (from the linked VOL or DSM tenant), the usage verdict, the source reads, the reads the indexers received (on distributed deployments, the indexers’ own record of the searches they ran, shown beside the search head’s figures, never added to them), the wildcard references, the write references (collect destinations, and the target index of a saved search’s summary indexing or Log event alert action), who read it, from which apps, when, and through which paths. Filters cover the usage, the path (its wildcard choice selects the wildcard references), the account class, the index name, the account and the app.

Index activity, the indexes of the last 7 days: cim_modactions, webserver, license_generator and trackme_summary searched, trackme_audit indirect only, with tiers, incoming data (VOL, VOL + DSM), source reads, indexers received, indirect and write references, who, apps, last search, via and the Detail and Open in Search actions

Detail opens the index. Source references gives the tiles — source reads, indirect and write references, the sourcetypes reported (best effort: a sourcetype never gets a verdict of its own), the volume per day and its cost, the reads the indexers received and the knowledge objects that declare it — and one row per account, app, sourcetype and path, with Fetch SPL examples to read the retained searches. Access timeline charts the named reads of the index over the period, clock-aligned, from the start of the collection.

Knowledge objects — declared against observed. Each published saved search, macro, event type, dashboard and data model, with the indexes its definition declares (wildcards expanded across the index catalog, shown muted) against the indexes its searches read in the period (an index read but not declared is outlined). Tiles count the states and filter the list:

  • In use — ran in the period and read the indexes its definition declares.

  • Ran without declaring — ran, but its definition names no index: what it read was resolved at run time.

  • Not run — declares indexes and did not run in the period, dated from when saved search runs began to be recorded.

  • Not due in this period — a scheduled search whose schedule leaves a longer gap than the period covers: it was not due, so it is not called Not run.

  • Disabled — disabled in its configuration.

  • Building block · used / not seen — a macro, event type or data model that searches went through, or that no observed search resolved directly.

  • Dashboard · viewed / not viewed — a dashboard whose panels ran searches in the period, or not.

Knowledge objects view: the coverage card, the state tiles (45 in use, 0 ran without declaring, 98 not run, 2 not due in this period, 12 disabled, 2 building blocks used, 64 not seen, 24 dashboards not viewed) and the object list with type, state, declared indexes, indexes read in the period, searches and last run

An object opens its detail: what it declares, what it read, how many searches ran and when.

Detail of the saved search uam_lab_report_macro, owned by lea.fontaine, In use: declares 3 indexes, read 3 in the period, 16 searches, the last one 28 minutes ago; license_generator, trackme_summary and webserver each declared by name and read 16 times

Evidence — what supports these numbers. The collection behind the figures: the fact records read, the collection runs and their outcome, the index catalog of each tier, the indexers’ confirmations, and the searches whose indexes could not be resolved, with the reason for each (a macro or saved search that cannot be found, a dynamic term, a wildcard role default) and the steps to resolve it.

Evidence view: the coverage card, the map query (no read limits reported), 3,363 indexed fact records, 80 runs reporting extraction, 3 incomplete runs, and the index catalogs by search head tier

An account’s data accesses. The account view has a Data accesses tab: the named reads of the account per index over the period, and Which data this account used, one lane per index with a marker for each group of nearby observations. Select a marker for the app, the usage origin and the claims behind it. An index view of the same timeline shows who used this index.

Account view of lea.fontaine, Data accesses: the named source reads per index (license_generator, trackme_summary, webserver) per 15-minute bucket since the start of the collection, and Which data this account used, one lane per index with the observed markers

Searches run as long as they need. Every view reads the evidence through your own Splunk search jobs, started when the view opens and followed by the UI: the loader names what is still being read (Still reading Collection runs…), and the searches are cancelled when you change the selection or close the view. Nothing is cut after a fixed time: a long period or a large tenant takes longer, and is complete when it is shown. Only a safety timeout of 10 minutes applies.

Settings → Data usage. Each search head tier is linked to the VOL tenant that gives its index volumes and to the DSM tenants that give its incoming data. Matching is automatic by default — the first VOL tenant and every DSM tenant you can see that is enabled and has the component, whatever account it collects through (2.4.23) — and you can override or remove a link. Indexes are matched by name: unselect a tenant that measures other indexers. A link grants no access: every read repeats the visibility check of the caller. The same section sets the apps whose reads are left out (trackme and Splunk_SA_CIM by default), whether internal indexes are shown, and the collection’s own limits.

UAM settings, Data usage section: Link data supply for the local search head with Automatically match supply tenants on, the volume tenant (VOL) and the data-source tenants (DSM) selected automatically, and the start of the collection and resolution limits

Note

What Data usage claims, and what it does not.

  • A read is a search that named an index — directly or through a macro, event type, saved search or data model that resolves to it. A wildcard is a reference, not a read: index=cribl_* does not prove which index supplied the events.

  • Each search is counted once. A search Splunk refused at its quota is not a read; a canceled or failed search that started is.

  • A sourcetype is reported on a best-effort basis and never gets a verdict.

  • The map covers what the audit trail of the indexing layer records and what the collection has seen so far. Until the collection covers the period, or seven days of it, nothing is called unread.

  • A declaration in a knowledge object is not a read, and a link to a VOL or DSM tenant is not evidence of a search.

The collection behind the tab — the Data usage report, its facts, the resolution limits and the REST operations — is described in Data usage.

How each kind of claim is processed (named reads, wildcards and their narrowing to the reader’s roles, searches the roles deny, definitions, index-less searches, late filters, writes, indexer confirmations, declarations, incoming data, verdicts and coverage), with diagrams, worked examples and limits, is the Data usage reference: how each claim is processed. The same reference opens in the product from every How this works link of the tab.

Activity

The raw evidence: Audit records or Correlated executions over a time range, a timeline split by account source, evidence status or account, account and app filters, and the 500 most recent records with an evidence inspector (Inspect). Executions are correlated from the audit records when you read them — the start and the completion of each search, the accounts and apps reported — and a notice says when an answer covers part of the range only.

Activity tab, the timeline split by account and the audit records table with account, app, search type, outcome and run time

Scheduled searches and the saved search view

The saved-search inventory per tier: tiles for the scheduled searches, the last full listing, the changes of the last run and the users and roles counts; the table with owner, owner class, schedule, enablement and first and last seen. Selecting a name opens the saved search view: its definition from the inventory, Fetch definition for the live definition, its runs with run time, CPU and peak memory, its findings and its change history, and a way to the owner’s account view.

Peak memory can read not measured for short searches that Splunk’s resource records never sampled. A saved search with the same name, app and owner on two tiers shares one set of run figures.

Policies

The built-in catalogue grouped by family (inventory, activity, resource, privileged) and your custom policies, shown as Cards, Compact cards or a List (remembered per user). Each policy shows its status, severity, account scope, thresholds (the effective values when they are derived from the capacity), exclusions, what it does on bank holidays and its last evaluation. Selecting the last evaluation opens the policy insight; Edit opens the policy editor. See Tuning the policies.

Audit changes

TrackMe’s own audit of this tenant: every configuration change, decision and collection request, with who did what and when — the same view every tenant type offers.

Collection health

Is UAM collecting what it should, and if not, why and what to do? Everything that says something is wrong comes with its explanation, its evidence and its action. Four views:

  • Status — one row per tracker that needs attention, from the same assessment the Configuration Guardian raises, so the page and the alert never disagree. A row expands to every reason with its meaning and where it was reported, What to do, the facts of the last run (status, finish, consecutive failures, search job, report and its live schedule, lock), the policies a run could not complete (with Show why), shortcuts to the evidence (Receipts, Tracker log, Search job, Report, Scheduler log) and Run now / Recover. Below, Problems in the last 24 hours lists the failed and incomplete runs of each tracker, recovered or not. Flow draws UAM’s own collection pipeline — the search head tiers and the indexing layer, the trackers, what they store and the policies — coloured by health.

  • Trackers — every tracker in one table: status, live schedule (read from Splunk), last run, last collected, duration, diagnostics, Run now (List now for a full listing) and Recover, filtered by name, family, status or tier. A row expands to its details: the activity checkpoint and the windows of the last run for Audit activity, the facts written, the claims and the unresolved searches for Data usage, the resource windows and the capacity for Resource usage, the login and automation scans for Account signals, the datasets of a catalog, the backfill progress and its controls (Pause, Resume, Cancel backfill, Plan again). For Audit activity, Data usage, Resource usage, Account signals and Policies, Searches of the last run lists each search the run sent, exactly as sent, with its time range, Copy and Open in Search. Custom policies have their execution health in their own family.

  • Data & storage — what UAM writes for the tenant and how to read it: the events per index and sourcetype (volume over a range, latest event), the KV collections (records, size, lookup, and a record viewer), the logs and every scheduled report with its live state, each with a shortcut.

  • Troubleshoot & help — the diagnostic glossary (every code UAM can report: what it means, where it comes from, possible causes and what to do; the codes this tenant reports now come first) and ready-made searches (tracker errors, failed or incomplete runs, skipped scheduled reports, REST calls, who changed UAM).

Every diagnostic code, wherever it appears, opens its explanation. A code is coloured as a warning or a failure only while its condition is current; a code that only describes a past, recovered run stays neutral. Only one run of a report happens at a time; a run whose search provably stopped is recovered automatically after five minutes, otherwise Recover stopped run releases it. A lock is never taken over on age alone.

The Configuration Guardian check uam_collection_health warns when collection is not running as configured — a missing or unscheduled report, missed schedules, repeated failures, a checkpoint that does not move, a range held for 15 minutes, windows or recounts waiting more than a day, an overdue listing, failing notification delivery, a custom policy that drifts from its schedule. See Configuration Guardian.

AI Agent

The UAM AI Agent: whether scheduled triage is on, its totals over 24 hours to 90 days, Run a review now, and the decision log with its filters (decision, mode, outcome, scheduled or asked). See The UAM AI Agent and the AI Assistant.

AI Agent tab: scheduled triage status, the totals (11 findings reviewed, 12 recommendations), Run a review now and the decision log of an inspect review, expected behaviour or true positive per Finding group

Settings

Every tenant setting in one page with one Save settings: Environment (the tiers and the route account), Activity scope (advanced), Collection (scheduled trackers, Store SPL search, resource collection), Indexes, Collection limits (time budgets, windows and cadences: there is no row limit), Notifications to operators, Notifications to users, Account classification, Data usage (the VOL and DSM links, the excluded apps and the resolution limits), Privileged users, AI Agent, Assignment and Finding actions. Changes apply from the next runs; Collect now applies them immediately. If someone else saved in the meantime, your save is refused and you are asked to reload. Every setting is described in Settings reference.

Accounts and classification

A finding about an account only makes sense once you know what the account is. UAM keeps three classes, and the class decides which policies apply:

  • human — a person. Detected automatically from interactive activity: a search run from Splunk Web (the Search app, a dashboard, a report opened by a person) or a browser login to Splunk Web. One is enough, and the class stays.

  • service — automation. Detected after seven distinct days of automation-only activity (API jobs, the scheduler, acceleration) and nothing a person does; withdrawn as soon as the account does what a person does. This detection needs the classic key=value _audit format, and the seven days must be inside the audit retention.

  • system — Splunk’s context identities (nobody, splunk-system-user) and the built-in admin, preset. System accounts are never evaluated by the account policies and never receive email.

On top of the detection, set manual classes per account (Users tab or Settings → Account classification) and ordered classification rules (exact names or wildcard patterns such as svc-*), with a preview of the effective class before saving. Precedence is always: manual class, system defaults, the first matching rule, human detection, service detection. An account nothing classifies stays unclassified and is left out of the policies scoped to human accounts.

Settings → Account classification: the detection switches, an ordered rule svc-* → service and the manual classes with their effective class

Tip

Classify the monitoring itself first. Classify the tenant owner (svc-a), your administrator accounts and other platform automation as system or service: they run TrackMe’s collection and the platform’s housekeeping and should not fill the findings. The trackme and Splunk_SA_CIM apps are excluded from the policies by default for the same reason.

Triage findings

Observed, not observed, and the status

A finding has two independent axes:

  • Whether the policy still sees the condition. While it does, the finding is observed. After a complete policies run that no longer sees it, the finding reads not observed — UAM never resolves it for you: a configuration or a behaviour that disappears from one evaluation still deserves a human decision. A run that could not cover everything (an incomplete inventory, an activity gap) may add findings, never clear them.

  • Your decision, the status: Open, Investigating, Resolved or Dismissed.

A resolved finding that is observed again is reopened. A dismissed finding stays dismissed until its dismissal ends — then it returns to open, so you get another look. The lifecycle shared with the other findings tenants is described in A finding and its lifecycle.

Finding groups

All the findings about one account and the saved searches it owns form one Finding group: the all-time searches and resource consumption of analyst-a together with the personal-schedule and real-time-schedule findings of the searches analyst-a owns. A saved search without an owner keeps a group of its own. You decide once, for the group: status, dismissal, assignee, notification snooze and severity apply to every member, and a finding that joins the group later takes its state.

Two UAM specifics:

  • Dismiss only this finding (member menu) sets one finding aside — for example a personal schedule you accept inside a group that also holds a real problem. It stays dismissed on its own while the rest of the group is decided on; Reopen this finding brings it back into the group’s state.

  • A new kind of finding on a dismissed group returns the group to open with the new finding, while the findings the dismissal covered stay dismissed: a dismissal never silences a new problem.

You can also group by hand — Merge into one Finding group (2 to 25 groups), Add to the Finding group (an existing manual group), Remove from the Finding group, Ungroup — for example to handle the accounts of one project together. The findings of one account always move together. See Finding groups.

Decisions

From the row, group and member menus, the selection toolbar (Bulk edit) and the Triage panel of the Investigate view:

Decision

What it does

Mark investigating

Someone is on it.

Resolve

The investigation concluded. The policy reopens the finding if it observes the condition again. Nothing is changed on the saved search or the account.

Dismiss

No action needed: for 1, 7, 30 or 90 days, or until reopened by hand (permanent). A dismissed finding stays listed under the Dismissed tile.

Reopen

Back to open; on a dismissed finding, ends the dismissal now.

Assign to me, Assign to…, Unassign

To a Splunk user or an assignment group. The assignee is kept when the finding reopens.

Snooze notifications

Holds the operator emails (from 1 hour to 30 days) without changing the status; Clear notification snooze resumes them. It does not hold the notifications to users.

Set severity

Overrides the policy severity (Low, Medium, High, Critical); the override survives later runs. Choosing the derived severity again (Members on a Finding group, Use policy severity on a single finding) restores it.

Notes

Markdown notes on a finding or a Finding group, permanent or expiring, plus a short investigation note recorded with each decision.

Exclude owner from this policy

Adds the finding’s owner to the exclusions of its built-in policy (custom policies write their exclusions in their SPL).

The row menu also has Send email communication, Send webhook… and your own actions under Custom actions, Ask the AI Agent…, the account view and saved search view, Copy finding ID and Open record / events in search.

Every decision is recorded in the history of the finding and of its Finding group, with the actor, the time and the note. Decisions are protected against concurrent changes: if the Finding group changed since you opened it, your decision is refused and the table reloads.

Assignment and findings actions

Assignment groups — “Platform team”, “Security operations” — and ordered routing rules (by policy, severity, subject type, owner or account, app, environment) are set in Settings → Assignment; routing assigns new Finding groups automatically, never overriding an existing owner, and can email the assigned team. See Assignment and automatic routing.

Findings actions (Settings → Finding actions) add drilldown searches, links, webhooks through system-wide Webhook accounts, and the email action to the finding menus, by hand or through per-policy automation rules (policy editor → Automations). UAM presets include Audit activity of this account, Scheduler runs of this saved search and Finding transitions. Tokens such as $subject.account$, $subject.owner$, $subject.app$ and $subject.name$ carry the finding’s values. See Findings actions and Webhook accounts.

Every finding transition is also indexed in the tenant’s notable index as sourcetype=trackme:uam:finding, ready for your own Splunk alerts, Enterprise Security correlations or SOAR playbooks.

Communicating with users

UAM can talk to two audiences: the operators who triage, and the accounts the findings are about. The two are independent.

Notifications to operators

Settings → Notifications to operators (off by default): the events that send an email (New finding, Finding reopened, Severity raised, Finding resolved, and optionally Finding dismissed and Finding assigned), the minimum severity (Medium), the policies (all), the email delivery account, up to 50 recipients, and the mode: a Digest (one email per policies run) or Per finding (one thread per finding). Emails are sent at the end of the next policies run, within about 15 minutes, and link to the finding. Send test email uses the saved settings, even while notifications are off.

Settings → Notifications to operators: events, minimum severity, policies, delivery account, recipients and the Digest / Per finding mode

Notifications to users

Settings → Notifications to users (off by default) emails the account a finding is about, so that people fix their own searches:

  • on a New detection, and when the Condition cleared (after two consecutive complete evaluations without it);

  • addresses come from Splunk’s user records on each tier, or from an administrator’s Override email address; system accounts never receive mail, human accounts do by default, service and unclassified accounts only if you opt in;

  • a common banner, introduction, footer, reply-to address and HTTPS support link, combined with each policy’s own subject and body (Policies → Edit → Email template);

  • limits: 50 emails per run, 3 per address per hour, 24 hours between repeat detections.

When you turn it on, UAM records the current findings as a baseline and only emails what is detected from then on, unless you choose Include current detections once. User emails follow the policy’s detection, not your triage: resolving, dismissing or snoozing a finding does not stop them.

Settings → Notifications to users: enablement, delivery account, account classes notified, common template (banner, introduction, footer, reply-to) and address coverage

Important

Before turning on notifications to users, check the address coverage shown in the section, set the reply-to address and the support link to your team, and preview the email template of each enabled policy. The people who receive these emails did not ask for them: make the first one clear and useful.

Send email communication

For a one-off message, Send email communication in a finding, member or Finding group menu prepares an email to the account concerned — one finding, or a whole group (up to 200 findings) in one email. You review the preview, can change the recipient once, and confirm explicitly. The email is sent by a TrackMe search job, never by the browser, and a delivery that could not be confirmed is reported, never resent blindly.

From a Finding group menu, the same dialog prepares one email for every finding of the group: here the three findings of jane.doe — a built-in activity policy, a custom policy and the CSV lookup footprint — under the subject 3 user activity findings.

Send email communication opened from a Finding group: one recipient, three included findings (a custom policy among them) and one combined email

The UAM AI Agent and the AI Assistant

The UAM AI Agent reviews findings the way an operator would: it reads the evidence TrackMe holds and writes its assessment as one current [AI Agent] note per finding.

  • Inspect changes nothing: assessment, confidence and recommendation.

  • Act also applies the decisions the tenant allows: mark investigating and dismiss for a time (on by default, never for a critical or high finding still observed), resolve a finding its policy no longer observes and classify an unclassified account (off by default). It never dismisses permanently, assigns, or changes a severity, a policy or a setting.

Ask the AI Agent… in a finding or group menu inspects one Finding group on demand; you can then let it act on that inspection. Scheduled triage (Settings → AI Agent, off by default) reviews up to 10 findings every hour — by default only critical and high ones, one per Finding group — and never overrides a decision or an assignment made by a person. Every decision appears under the actor AI Agent in the history and in the AI Agent tab. The AI Agent needs the AI Assistant to be enabled with at least one AI provider; the scheduled triage also needs a tenant owner other than nobody or splunk-system-user. The UAM AI Agent replaces the UAM Findings Advisor of earlier releases. See Findings AI Agents and AI Agents for findings.

Custom instructions. The tenant administrators can give the agent instructions in plain English (Settings → AI Agent → Custom instructions, also in Manage AI Agents automation and in the Configure tenant dialog): at most 4,000 characters, line breaks kept. Every run of the agent follows them, asked or scheduled — for example, “The accounts named svc_* are service accounts.” They never widen what the agent may do: the allowed decisions and TrackMe’s checks before every decision still decide. The tenant’s shared Custom Instructions of Manage AI Agents automation are different: they apply to scheduled runs only.

Evaluate… tries the unsaved draft before you save it: the agent runs in inspect mode on one open finding, or on a review of the open findings (at most 10, the most severe first), optionally side by side with a run without custom instructions. An evaluation writes nothing — no note, no decision, no review record — and is recorded with the status evaluation. See AI Agents for findings.

Ask the AI Agent result on a finding: assessment, confidence, recommendation and the Act button

The AI Assistant of a UAM tenant knows the workspace, the policies and the open findings: ask it to explain a finding, compare accounts or suggest what to tune, and it can propose an AI Agent review or a triage action for you to confirm. See AI Assistant & Concierge.

AI Assistant on the UAM Tenant Home answering a question about an open finding and proposing a next step

Tuning the policies

The built-in catalogue

Sixteen policies in four families. Seven are enabled on a new tenant; the four privileged policies are off until you turn them on.

Policy

Family

Default

What it detects

Scheduled search owned by a human account (personal_schedule/v1)

inventory

on, Low

An enabled schedule owned by a person: it stops when that account goes.

Scheduled search owner is not classified (unclassified_schedule/v1)

inventory

off, Medium

An enabled schedule whose owner is neither human nor service.

Scheduled search exceeds the hourly schedule limit (dense_schedule/v1)

inventory

off, Medium

A cron with more minute slots per active hour than the limit (12).

Scheduled search has an all-time dispatch lower bound (all_time_schedule/v1)

inventory

on, Medium

A schedule whose earliest time is 0 or empty.

Scheduled search uses a real-time dispatch range (realtime_schedule/v1)

inventory

on, Medium

A real-time scheduled search.

CSV lookups attributed to an account exceed its size allowance (csv_lookup_footprint/v1)

inventory

on, Medium

Per tier, one CSV lookup over the file limit (100 MiB) or an account’s CSV lookups over its total (500 MiB), attributed to the probable writer, else the recorded owner.

Account launched an unusual number of searches (launch_burst/v1)

activity

off, Medium

More distinct searches in the policy window than the limit (200).

Account ran repeated long searches (costly_search/v1)

activity

off, Medium

More than 3 searches of at least 300 seconds.

Account ran all-time searches (all_time_search/v1)

activity

on, High

Any all-time search (the maximum is 0).

Account’s search resource consumption exceeded its allowance (resource_consumption/v1)

resource

on, Medium

CPU over the allowance in the window (10 % of the search capacity by default); the severity is raised one level when the account dominates the environment (40 %).

Account ran repeated resource-intensive searches (resource_intensity/v1)

resource

on, Medium

More than 3 searches heavy in CPU or memory.

Account’s CPU consumption is unusual for its daily baseline (resource_deviation/v1)

resource

off, Medium

Today’s CPU above a multiple (3) of the account’s daily 95th percentile over 14 days. Ships in Preview mode: it shows what it would match, without raising findings or sending notifications, until you turn the mode off.

Account became privileged (privileged_account_new/v1)

privileged

off, High

An account that became privileged: created privileged, given a privileged role, or a role it holds now imports one. Reported once per change, with who made it when Splunk audited it.

Role gained a privileged capability (privileged_role_capability/v1)

privileged

off, High

A role whose own capabilities or imported roles changed so that it grants more privileges, or a new role created privileged. One finding about the role, listing the roles that inherit it and the accounts newly privileged through them.

Privileged account is dormant (privileged_account_dormant/v1)

privileged

off, Medium

A privileged account with no search, no Splunk Web login and no successful login reported by Splunk for 30 days.

Privileged accounts differ from the reference (privileged_reference/v1)

privileged

off, High

A privileged account your reference (typed names and lookup tables) does not declare; optionally a declared account that holds no privilege or no longer exists (Low).

The activity and resource policies judge 15-minute windows of collected evidence by default. A window ends where the collected evidence ends, never at the current time, and consecutive windows always touch: no minute is skipped, and activity collected late is evaluated once it is collected. The inventory policies follow the catalog cadence (the full saved-search listing is daily by default). The resource policies stay learning — no finding, nothing cleared — until the tenant holds 24 hours of resource history and a measured capacity. The full catalogue, with every threshold and its bounds, is in The built-in policy catalogue.

Editing a policy

Edit opens the policy editor:

  • General settings — status, During bank holidays (see Bank holidays), severity, account scope (human accounts only, classified accounts, all accounts), thresholds (the resource policies can derive them from the measured capacity, auto thresholds), excluded owners and excluded apps (exact names, up to 500), and Preview policy to see what the draft would raise against the current data before you save.

  • How it is evaluated — the rule in plain words, then the exact searches it runs (built from the saved settings, with their time range, Copy and Open in Search), or, for an inventory policy, the records it judges.

  • Email template — the subject and body of the user emails for this policy, for a new detection and for a cleared condition, with a preview and a test delivery; Restore policy default brings back the shipped text.

  • Actions and Automations — the findings actions offered on this policy’s findings, and the automation rules that run a webhook when they change.

The policy insight (from the last evaluation of a policy) explains a run: the verdict, what the run saw, what was matched, excluded, not applicable or could not be judged and why, the owners behind the unknowns (classifiable in place), the exclusions and a seven-day history, with the runs of those seven days that could not judge everything and their reasons (kept after the problem recovered). It is the first place to look when a policy raises nothing, or too much.

Policy insight of Account ran all-time searches: a complete evaluation (3 evaluated, 3 matched), what the run saw, the seven-day history per policies run and the excluded apps

Privileged policies

Four policies watch who holds high privileges, from the assessment of the Privileged users tab (its rules: Settings → Privileged users). They are off by default: turn on the ones you need in Policies.

  • Account became privileged reports, once, an account that became privileged on a tier: a new account created privileged, a privileged role assigned to it, or a role it holds that now imports a privileged role. When Splunk audited the change, the finding names who made it and when; a role mapped by a SAML or LDAP provider has no such record.

  • Role gained a privileged capability reports, once, a role whose own definition changed so that it grants more: its own capabilities, the roles it imports, or a new role created privileged. Inheritance is transitive, so one change reaches every role that inherits it, directly or through other roles: the finding is about the role whose definition changed, and lists the roles that inherit it and the accounts newly privileged through them — one change, one finding.

  • Privileged account is dormant reports a privileged account with no activity for 30 days (configurable): no search, no Splunk Web login, no successful login that Splunk reports on any tier. It judges an account only once it has been in the inventory and the tenant’s evidence covers the whole period.

  • Privileged accounts differ from the reference compares the privileged accounts with a reference you maintain — the accounts declared privileged, optionally per search head tier. It reports a privileged account the reference does not declare (High) and, if you keep Report stale references on, a declared account that holds no privilege or no longer exists (Low).

The reference is set in the policy editor, from two sources that add up, case-insensitive:

  • Account names typed in the policy (up to 1,000), declared on every search head tier; * matches any text (svc-admin-*).

  • Lookup tables (up to 20): Add a lookup table picks the Splunk deployment (the local search tier, or a remote deployment through a TrackMe remote account), the lookup (a CSV or KV store lookup shared with the TrackMe app, with a preview of its first rows), the account field, an optional tier field (the search head tier a row declares the account for: local or a remote account name; not set: every tier) and the match mode — Exact (a value holding * is ignored) or Wildcard. Each lookup holds up to 500,000 values and is read at every policies run; a lookup that cannot be read, or reads empty, keeps its last good values, and one never read yet makes the policy judge nothing until it can be read. TrackMe’s own lookups are not offered.

Test reference reads every lookup now, as a search, and shows what each source declares, the privileged accounts the policy would report as undeclared, the stale references and the patterns that declare nothing. It saves nothing.

The first evaluation of a change policy records the current state and reports nothing, and a change of the Privileged users rules records the new state without reporting. The accounts Splunk operates on a Splunk Cloud stack (their email address is support@splunk.com) are left out unless you include them. These policies never email the account a finding is about; notifications to operators work as for every policy. A role finding has no account to write to: Send email communication is not offered on it.

Policies tab, the four privileged policies as cards: Account became privileged, Role gained a privileged capability (subject role), Privileged account is dormant (7 days without activity) and Privileged accounts differ from the reference (its declared accounts and lookup tables), each enabled with its severity and last evaluation

Custom policies

When the catalogue does not cover a rule of your own, write it in SPL: Policies → Create policy (TrackMe administrators of the tenant).

  1. Definition — title, description, default severity and remediation.

  2. Search and schedule — the search source (an account policy reads through the route account), the app context, the SPL (up to 64 KiB), a relative window ending now (at most 31 days) and a five-field cron.

  3. Map search results to findings — the subject type, Account or Saved search (owner, app, name), and which result fields carry the identity, dimensions, evidence, count and time.

  4. Preview policy — runs the draft as you and shows up to 20 candidate findings. A preview with no results cannot verify the mapping.

  5. Save draft, Publish (an immutable revision), Enable (a scheduled search for the published revision). Version history lets you load, compare, restore or delete versions; Disable, Archive and Delete permanently end a policy (its findings, evidence and notes remain).

The definition also has a Bank holidays section (see Bank holidays). Custom findings appear with the built-in ones, join the Finding groups and are triaged the same way. In this release, operator and user notifications cover the built-in policies only, and a custom policy writes its exclusions in its SPL. A tenant holds up to 100 custom policies, 25 of them enabled; their execution health is in Collection health → Trackers, family Custom policies.

Bank holidays

A UAM tenant uses the bank holiday calendars like every tenant: the system default calendars, unless it selects its own or none (Manage Bank Holidays → Tenants & settings, or Configure tenant → Bank Holidays). One effect applies to it, Suppress new alerts, shown as Hold new findings and on by default: while a holiday of the tenant’s calendars is active, the policies set to hold open no new finding. SLA, monitoring windows and ML training do not apply to a UAM tenant.

Each policy chooses in its editor, under During bank holidays: Hold new findings or Keep opening findings. The defaults:

Holds new findings

Keeps opening findings

Scheduled search owned by a human account, Scheduled search owner is not classified, Scheduled search exceeds the hourly schedule limit, Scheduled search has an all-time dispatch lower bound, Scheduled search uses a real-time dispatch range, CSV lookups attributed to an account exceed its size allowance, Privileged account is dormant

Account launched an unusual number of searches, Account ran repeated long searches, Account ran all-time searches, Account’s search resource consumption exceeded its allowance, Account ran repeated resource-intensive searches, Account’s CPU consumption is unusual for its daily baseline, Account became privileged, Role gained a privileged capability, Privileged accounts differ from the reference

The inventory policies and the dormancy policy are reviews: nobody reviews them on a day off, and they are still there the next working day. A privilege granted on a day off is more telling, not less. The activity and resource policies keep opening findings, because unusual activity on a day off is more telling, not less. A configuration saved before TrackMe 2.4.19 reads each policy’s default.

When a policy holds:

  • An inventory policy holds while a holiday is active at the time of the run; an activity or resource policy holds when a holiday overlaps its activity window, so a burst on the last evening of a holiday is still a holiday burst.

  • A held match writes nothing: no finding, no Finding group, no notification to operators or users, no findings action, and it uses none of the findings capacity. The first policies run after the holiday opens the finding if the condition is still there.

  • A finding already observed is evaluated as on any other day: updated, escalated, or no longer observed. A held match does not reopen a resolved or dismissed finding.

  • If the calendars cannot be read, nothing is held.

A custom policy keeps opening findings unless its definition turns the setting on (Bank holidays section of the custom policy editor; it takes effect when published). It holds while a holiday is active at the end of its search window.

Where to see it. Each policy on the Policies tab says During bank holidays: holds new findings or keeps opening findings. A policy that held matches shows held N (bank holiday) under its last evaluation, and its policy insight says Held N (bank holiday: <names>) with a Bank holidays line for the run. The policies run receipt records the state (holding, off or unavailable), the holiday names, the matches held and the enabled policies set to hold; a custom policy run records what it held the same way.

UAM Policies tab: each policy shows During bank holidays, holds new findings for the scheduled search configuration reviews and keeps opening findings for the account activity policies

Upgrading to TrackMe 2.4.21

  • Three trackers for the indexing layer. What the activity tracker did in one run is now Audit activity, Resource usage and Account signals, each with its own schedule, time budget, lock and health. The two new reports are created automatically within five minutes of the upgrade, with the tenant’s owner; they continue where the earlier tracker stopped, and nothing is collected twice.

  • No row limits any more. The seven record-count settings of Collection limits are removed: what a run cannot finish in its time continues at the next run. Stored values are ignored and disappear at the next save; an API client that still sends them keeps working.

  • New time budgets. Every run budget defaults to its tracker’s interval less 30 seconds. A budget still at its earlier default moves to the new one; a value you chose is kept. A scheduled run dispatched late ends 30 seconds before the next scheduled run, so the scheduler never skips one.

  • Correlated executions are no longer indexed. They are correlated from the audit records when they are read. Your own searches, dashboards, alerts or custom policies on sourcetype=trackme:uam:execution return nothing for activity after the upgrade: move them to sourcetype=trackme:uam:observation. The UAM events also carry fewer fields (see The evidence).

  • Backfill. A new tenant starts without backfill unless you choose a range, in 5-minute buckets. An existing tenant keeps its bucket size; set Backfill bucket (s) to 300 to move it to 5-minute buckets (the coverage already collected is kept).

  • Splunk restart. Some index-time settings of the UAM events take effect after the splunkd restart of the upgrade, on the instances that parse the UAM log.

  • Privileged users and privileged policies. The Privileged users tab and its settings appear with the release defaults (*admin*, trackme_admin excluded). The four privileged policies are off; when you turn on a change policy, its first evaluation records the current state and reports nothing. The users catalog now also collects the last successful login Splunk reports (never an inventory change).

Upgrading from TrackMe 2.4.18

  • The first policies runs gather the existing findings into Finding groups; decisions, notes and history are kept.

  • The inventory moves to the REST API catalogs: each tier gets its new knowledge-object reports, and its first full listing runs within the hour. The users and roles reports move from a 5-minute to a 15-minute schedule.

  • The UAM Findings Advisor is replaced by the UAM AI Agent; an enabled advisor’s settings are carried over once.

Where to go next

See also