User Activity Monitoring — reference

This page is the reference companion of User Activity Monitoring: the tenant model and the search head tiers, every report and its schedule, the knowledge-object catalogs, the historical backfill, the indexed evidence, identity and classification, privileged users, the catalogue of built-in policies, custom policies, the findings lifecycle and Finding groups, notifications and communications, the UAM AI Agent, Collection health and the Configuration Guardian check, every setting, the roles and licence, the REST endpoints, the upgrades and the troubleshooting table.

For the task-oriented guide (creating a tenant, reading the workspace tab by tab, triaging a finding), read User Activity Monitoring. The concepts User Activity Monitoring shares with the other findings tenants — Finding groups, triage, assignment, notifications, findings actions, Webhook accounts and the AI Agents — are explained once in Findings, Finding groups and findings actions. For a narrated end-to-end walkthrough, read the User Activity Monitoring white paper.

The tenant model

User Activity Monitoring (UAM) is a dedicated tenant type, created from the User Activity Monitoring card of the tenant creation wizard:

• A tenant is either an entity tenant (DSM, DHM, MHM, FLX, FQM, WLK, VOL) or a UAM tenant, never both. Creation refuses a mix (uam_tenant_must_be_dedicated), the Manage components menu is disabled on a UAM card, replica trackers are refused and the add / remove component endpoints answer uam_tenant_is_dedicated.

• There are no entities: no decision maker, no impact score, no green / orange / red state, no stateful alerting, no priority / SLA / logical group. The unit of work is the finding, triaged as a Finding group.

• A UAM tenant watches one indexing layer seen through one to ten search head tiers: the accounts (human, service or system) and their search activity, their resource cost (CPU, peak memory, share of the environment) and their knowledge objects; the saved searches and the other knowledge objects of each tier; and the policies — sixteen built in, plus the custom policies you write — that raise findings about them.

• Evidence lives in indexes, state in the KV store. Collection runs in scheduled searches, never in a REST handler.

• The tenant identifier of a UAM tenant is short — lowercase letters, digits and hyphens, up to 20 characters — because it keys the per-tenant collections and reports.

• The Virtual Tenants card shows the open and high / critical findings and the collection problems; a double-click opens a read-only overview. Tenant Home opens the UAM workspace (tabs Findings, Users, Privileged users, Activity, Scheduled searches, Policies, Audit changes, Collection health, AI Agent and Settings, described in User Activity Monitoring).

Note

What UAM claims, and what it does not. An account is the principal reported by Splunk in the audit records — not a verified person. Distinct search ids are a launch-count proxy. Collection is bounded by time only and resumable: nothing is dropped to fit a row count, and what a run cannot finish continues at the next run. Coverage still depends on what the sources retain and what the collection accounts can see: gaps and partial coverage are recorded and shown (receipts, Collection health), never hidden. There is no machine learning: the daily CPU deviation rule compares against a percentile baseline.

Per-tenant storage

KV collection	Content
kv_trackme_uam_settings_tenant_<tid>	The configuration, the checkpoints, the latest receipt and the lock of every job, the capacity, the detection records, the notification outbox, the user notification state and the private contact store.
kv_trackme_uam_inventory_tenant_<tid>	The policy inventory: one row per saved search, user, role and CSV lookup file, per tier, with present, first / last seen and the owner’s class.
kv_trackme_uam_knowledge_tenant_<tid>	The published knowledge-object catalogs and their definitions (administrator access only, no lookup).
kv_trackme_uam_findings_tenant_<tid>	One row per finding, with its investigation state and its Finding group.
kv_trackme_uam_finding_groups_tenant_<tid>	One row per Finding group (automatic or manual).
kv_trackme_uam_custom_policies_tenant_<tid> / kv_trackme_uam_custom_policy_state_tenant_<tid>	Custom policy definitions (drafts, published revisions) and their execution state.
kv_trackme_uam_actions_tenant_<tid>	Findings actions definitions and automation rules.

The settings, inventory and findings collections each have an administrative lookup trackme_uam_<kind>_tenant_<tid>. Two global collections serve every UAM tenant: kv_trackme_uam_agent_decisions (the UAM AI Agent decision log, 90-day retention) and kv_trackme_uam_custom_execution (custom policy execution slots).

Search head tiers and the route account

One indexing layer, one to ten search head tiers:

Data	Lives on	Collected
Saved searches, users, roles, lookup files and the other knowledge objects	each search head tier	once per tier, from the Splunk REST API
_audit activity, _introspection resource usage, search completion lines	the indexing layer (every tier forwards its internal indexes there)	once per tenant, through the route account

• Search head tiers (tiers) — 1 to 10 unique names: local (the search head TrackMe runs on) and / or TrackMe remote accounts (see Remote Splunk deployments), one account per tier. Every selected tier must feed the same indexing layer; that invariant is the administrator’s responsibility. The wizard tests every remote tier and waits until each is reachable.

• Route account for indexing-layer searches (account) — the tier through which activity, resources, capacity and the audit lookups are read: local when selected, else the first tier. It must be one of the tiers.

• A user present on several tiers is one account (roles united, tiers listed); a saved search belongs to its tier. Each tier has its own reports, locks and receipts, so a slow tier never starves another and a skipped tier is visible on its own.

• Typical shapes: on Splunk Cloud, TrackMe on the ad hoc search head (local) plus a remote account for a premium-app search head; on Splunk Enterprise, any number of remote accounts over the same indexers; for central monitoring, remote accounts only.

• A search head that keeps its internal indexes local (no forwarding of _audit / _introspection) hides its searches, logins and resource usage from the tenant — the forwarding prerequisite applies to every signal alike.

Activity scope (advanced)

The tiers select REST connections for the inventory; they do not restrict which events are read in a shared indexing layer, so a search head that is not a tier still contributes activity, classification evidence and resource cost. Activity scope (search_head_scope, wizard and Settings) restricts the activity to search head hostnames:

• include and exclude — up to 20 hostname patterns each (1 to 253 characters), matched case-insensitively on the whole hostname; * matches any characters, ? one character, everything else is literal. No SPL, no regular expression.

• Both empty (the default): every origin. An empty include list means every origin except the exclusions. Exclusions win.

• Include every member of a cluster. Excluding only the cluster members does not exclude a management node that also runs searches; to isolate one search head tier completely, include its hostnames.

• The scope applies to the live collection and to the backfill: audit records, logins, automation evidence, resource records (indexer samples are mapped back to the search head of their search; an ambiguous mapping is excluded and reported as incomplete coverage), completion CPU and the capacity’s search head part. A change starts a new scope.

• The scope is written into the base searches, so the indexers return only what can be in scope instead of every search head’s events. The audit, login and automation reads filter on host (indexed metadata, matched case-insensitively); the resource and capacity reads keep the in-scope search heads’ samples and the indexer processes of their searches (remote_<search head> origins), so the capacity lists the indexers that serve the scope. The base-search terms always select at least what the exact rule keeps. An exact name or a name ending in * is the filter itself. An include holding ? or an inner * is written on its literal prefix (team-*.example.net → team-*; an include without one, such as *.example.net, leaves the includes out of the base search), an exclude of that kind is not written in the base search, and the exact rule is applied after the base search, so nothing in scope is ever skipped. Patterns written as exact names or with a trailing * filter the most at the source. A tenant without a scope runs the same searches as before.

• It is a monitoring rule, not an access-control boundary: nothing is deleted from the source indexes.

Reports and schedules

Provisioning creates every report of the tenant and schedules them at once. Each report runs | trackmeuamtracker tenant_id=<tid> job=<job> (or its own command) as the tenant owner, with its own lock, its receipt (the outcome of the latest run, kept in KV and indexed as trackme:uam:receipt) and its entry in the tenant’s operational status.

Report	Schedule	Does
trackme_uam_activity_tracker_tenant_<tid> (Audit activity)	*/5 * * * *	The audit trail from a durable ingestion checkpoint through the route account, the delayed recounts and re-reads of its ranges; CSV writer hints.
trackme_uam_resources_tracker_tenant_<tid> (Resource usage)	2-59/5 * * * *	The CPU and memory of every search, window by window from its own checkpoint, their re-reads, and the hourly search capacity. Unscheduled while resource collection is off.
trackme_uam_signals_tracker_tenant_<tid> (Account signals)	9-59/15 * * * *	The Splunk Web logins and the automated searches of the audit trail, up to where the audit activity has read, for the human and service account detection.
trackme_uam_data_usage_tracker_tenant_<tid> (Data usage)	4-59/15 * * * *	The indexes each audited search names, from its own checkpoint in the audit trail, resolved through the published catalogs of the search’s tier; the indexers’ confirmations on distributed deployments. See Data usage.
trackme_uam_policies_tracker_tenant_<tid>	3-59/15 * * * *	Evaluates the built-in policies, writes findings and Finding groups, expires temporary dismissals, delivers operator and user notifications, runs the automation rules.
trackme_uam_inventory_tracker_tenant_<tid>_tier_<tier>	*/15 * * * *	Summarises the published catalogs of the tier for the policies run (reads no source).
trackme_uam_listing_tracker_tenant_<tid>_tier_<tier>	hourly wake (37 * * * *)	Lists every saved search of the tier when due: every Catalog collection every hours (24) inside the off-peak window (01:00 for 5 hours, search head local time). A tier’s first listing runs at the next hourly wake, whatever the window. A listing longer than its budget continues at the next wake its window allows.
trackme_uam_kos_users_tracker_… / …kos_roles…	15-minute wake	Users (with the email address for user notifications) and roles, read every Users and roles every minutes (60).
trackme_uam_kos_lookup_files_tracker_…	15-minute wake	CSV lookup file metadata (size, owner, modification time; never the contents), read every 15 minutes.
trackme_uam_kos_<family>_tracker_… × 10	hourly wake, each at its own minute	The ten other knowledge-object families, on the catalog cadence, in the off-peak window.
trackme_uam_backfill_tracker_tenant_<tid>	17 * * * *	The history before the live start, bucket by bucket; retires itself when done.
trackme_uam_custom_<id>_tenant_<tid>	the policy’s own cron	One per enabled, published custom policy.
trackme_uam_agent_tracker_tenant_<tid>	hourly, at a minute derived from the tenant	The UAM AI Agent’s scheduled triage; created unscheduled, scheduled only when AI is available and the tenant enables it.

The per-tier catalog reports are named trackme_uam_kos_<dataset>_tracker_tenant_<tid>_tier_<tier>. The tracker schedules are fixed; the cadences inside them (Catalog collection every, Users and roles every, the listing window) are settings.

Run budgets

Time is the only bound of a run: what a run cannot read in its time continues at the next run through its checkpoint, never cut to fit a row count. Every budget defaults to its tracker’s interval less 30 seconds, the margin covering the run’s own writes before the next scheduled run:

Tracker	Setting (key)	Default
Audit activity	Audit activity max seconds per run (activity_max_seconds)	270 s (30–900)
Resource usage	Resource usage max seconds per run (resources_max_seconds)	270 s (30–900)
Account signals	Account signals max seconds per run (signals_max_seconds)	870 s (30–900)
Data usage	Run budget in Settings → Data usage (data_usage_max_seconds)	870 s (30–14,000)
Saved-search listing and catalogs	Full listing budget (inventory_listing_max_seconds)	3,570 s (300–14,000)
Historical backfill	Backfill max seconds per run (backfill_max_seconds)	3,570 s (600–3,570)

No skipped runs. The scheduler may dispatch a run up to five minutes late, and it skips a run whose previous run is still going. A scheduled run’s budget is therefore capped to end 30 seconds before the tracker’s next scheduled run; a run dispatched so late that less than 15 seconds would remain records a skip (dispatched_after_schedule_cutoff) instead of starting. The receipt records the budget applied (run_budget: the seconds, whether they were capped and by how much the dispatch was late), and the tracker’s Duration in Collection health explains a cap.

Each indexing-layer tracker keeps a share of its own budget for its re-reads, recounts or capacity refresh only when some are due; nothing is shared between trackers, so a slow resource read never takes time from the audit trail, and the other way round.

Collect now, Run now and manual collection

• Collect now (workspace status strip) runs every tier’s inventory summary, then the audit activity, the resource usage and the account signals, then the policies, and every tier’s full saved-search listing. The other catalog families record a request their reports pick up at their next wake.

• Run now / List now (Collection health) dispatches one tracker.

• With Scheduled trackers off, nothing runs on a schedule: collection only happens on request. Scheduled collection requires a tenant owner (scheduled_collection_requires_tenant_owner).

• A scheduled run never blocks the workspace. A Collect now refused because a run holds the lock (uam_job_running_or_recovery_required) is explained, not reported as an error.

Locks and recovery

One run per tracker at a time. Locks carry the local server GUID and the search id and are never stolen by age alone:

• A lock at least five minutes old whose search job has provably stopped on the same search head is recovered automatically at the next acquisition.

• Recover stopped run (Collection health, POST recover) does the same on demand.

• Locks held by another cluster member, locks without a search id and configuration locks without job ownership are never recovered automatically: verify the originating process on its search head.

• The tenant lifecycle (delete, enable, disable, RBAC) has its own guard bound to the process that holds it, so a request whose process died never blocks the tenant for good.

Knowledge-object catalogs

The inventory is read from the Splunk REST API only — no _internal or _audit search is used to discover objects or changes. Each dataset of each tier is its own scheduled traversal:

Family (account view tab)	Datasets	Cadence
Saved searches & alerts	Every saved search (reports and alerts), with owner, app, sharing, cron, dispatch bounds, enablement and the definition.	Catalog collection every (24 h), off-peak
Users and roles	Users (with their email address, kept in a private store) and roles.	Users and roles every (60 min)
Lookups	CSV lookup files (metadata only), lookup definitions, automatic lookups.	Files every 15 min; the others on the catalog cadence
Dashboards	Dashboards and views.	Catalog cadence
Macros	Macros.	Catalog cadence
Event types	Event types.	Catalog cadence
Fields	Field extractions, field transforms, field aliases, calculated fields.	Catalog cadence
Tags	Tag assignments.	Catalog cadence

• Every traversal is paged (Inventory page size) and bounded by time (Full listing budget), never by a record count, and publishes a complete generation or nothing. A traversal that does not finish within its budget continues at the next run of its tracker, from where it stopped, in a staging generation, and is published once complete; until then the previous catalog stays in use and keeps the policies complete. Collection health shows such a run as collected with Listing continues next run; Listing made no progress, Listing restarted (access refused) and No page read in time are the warnings, and a listing carried over for more than seven days starts over. A failed traversal keeps the last good catalog. Only one traversal runs at a time per tenant and tier; a busy report waits for its next wake.

• Definitions — saved search SPL, macro bodies, event type searches, dashboard source, field expressions, lookup configuration, tag assignments — are kept up to 256 KiB per object in the administrator-only knowledge collection, from an explicit attribute list (never credentials, never CSV contents). They are not executed, indexed or copied to the policy inventory. Older generations (other than the current and the previous one) are deleted after 30 minutes.

• CSV lookup sizes before Splunk 10.4. The lookup file listing gives no size before Splunk 10.4 (Splunk 10.4 and later report it, with no extra access); UAM then reads the size and modification time of each CSV lookup file listed without one through admin/file-explorer, one lookups directory at a time (the path only locates the file and is never stored). The tier’s collection account must be allowed to read that endpoint, which by default only admin and splunk-system-role can: a least-privilege account cannot retrieve CSV lookup sizes before Splunk 10.4. A refusal is reported as lookup_size_access_denied (Lookup sizes refused), never as a partial listing.

• Each saved search, user, role and lookup file is diffed against its previous row: a new row is added, changed fields changed (with changed_fields), a classification change alone reclassified, a row seen again after removal reappeared; a row absent from a complete traversal is marked present=false — never deleted. A partial traversal marks nothing removed. Each change is indexed as trackme:uam:inventory.

• The catalogs reflect what the tier’s account can see: coverage of private objects depends on its permissions. A traversal is not an atomic snapshot.

• Where they appear: the account view’s owner catalogs (Saved searches & alerts, Dashboards, Macros, Event types, Fields, Lookups, Tags), Search knowledge objects on the Users tab (across every owner), the saved search view, and Collection health → Trackers (family Knowledge object catalogs). Tenant administrators only.

Activity

The Audit activity tracker reads the _audit trail from a durable ingestion-time checkpoint. Time is the only bound of a run: nothing is cut to fit a row count, and what a run cannot read in its time continues at the next run.

1. It loads the checkpoint, or initialises it at now − window on the first run or when the environment changed (checkpoint_reset on the receipt).

2. It reads ranges of up to Activity window (300 s) from the cursor, less the part of the Activity overlap still owed, until it catches up (now − 10 s) or its budget (Audit activity max seconds per run, 270 s) is spent. Both audit formats are evidence: the legacy Audit:[…] key=value line and the structured JSON record. The rows are normalised and correlated by search id.

3. A busy range is split, never cut. A range holding more audit lines than one read returns, or that Splunk finalized, truncated or reported with dropped results, is halved, down to one second, and a one-second range is read whole, streamed in ingestion order.

4. A range cut by the run’s budget resumes at the next run after the last line it wrote (resume_next_run); near the live edge it is read again from its first line instead, so a line indexed a moment late is not missed.

5. A failing range is held and retried, never skipped (window_held): it is retried at once within the run, then at the next runs, and halved after three failures to isolate the failing span. A range held for 15 minutes is a Collection health problem.

6. Indexers that do not answer. A range read while some search peers did not answer keeps what the others returned and is listed to be read again once every peer answers (window_reread_pending), with a back-off of 5 minutes up to an hour. The re-read writes only the lines missing the first time.

7. Late lines are recounted. About ten minutes after a range is closed, it is counted again with the same search; more lines than were written lists the range for a re-read (window_recount_mismatch).

8. The overlap and a ledger of what was written make sure nothing is written twice, resumed, retried and re-read ranges included. The checkpoint is persisted after every range.

9. Splunk’s search messages are recorded, never interpreted: only a clear failure (a failed or cancelled job, a broken read) discards a search. Evidence-quality diagnostics (normalization_gap, locator_conflict, invalid_event_envelope, unsupported_source_contract) are recorded but never block.

The same run records the CSV lookup writes it saw (probable writer hints for the CSV policy). The resource usage, the capacity and the login and automation scans are trackers of their own (below).

Account signals

The Account signals tracker (every 15 minutes, Account signals max seconds per run, 870 s) scans the audit trail for Splunk Web logins and automation evidence (searches run by the scheduler, the REST API, acceleration), the signals behind the human and service detection (see Data usage). Each scan has its own cursor and trails the audit activity’s cursor without ever holding it; the login scan takes up to half of the budget, the automation scan the rest. A scan range that cannot be read is listed to be read again, like an audit range. Before the first audit activity run, the tracker waits (account_signals_wait_for_activity).

Source contract. The audit source contract is the format, never the Splunk version: a line that parses as the legacy key=value grammar or the structured JSON is evidence. The qualified families are Splunk Enterprise 9.4, 10.4 and 10.6 and Splunk Cloud 10.5; any other version is read all the same and only adds the soft diagnostic source_version_unqualified. With Source version assumption on (the default), emitters whose version could not be verified are parsed under the connected head’s version and labelled assumed.

Resource usage and CPU accounting

With Collect search resource usage on (the default), the Resource usage tracker (every 5 minutes, two minutes after the audit activity, Resource usage max seconds per run, 270 s) reads the resource records window by window from its own checkpoint and writes one trackme:uam:resource record per search and account: CPU seconds, the part spent on the indexers, peak memory, elapsed time, node count, and Splunk’s own search type, provenance, mode and label. Its reads are paged and resumable like the audit’s: every search of a window is kept, a window read while peers did not answer is collected again and replaces the earlier collection, and the policies never count a replaced collection. With resource collection off, the report is unscheduled and not assessed.

CPU follows Splunk’s own accounting where it exists, decided per window and per tier (search head, indexers) from the data:

• Splunk Cloud 10.5 accounts CPU at search completion (the audit cpu_time on the search head, the indexers’ completion lines). A tier with a completion figure counts its CPU once, at completion.

• A tier without one keeps the CPU integrated from the _introspection per-process samples (Splunk Enterprise 10.4).

• Every record carries cpu_source (completion, head_completion, peer_completion or introspection). Real-time searches always use sampled CPU.

• Peak memory reads not measured for a search no introspection sample caught (short searches in pooled processes on Splunk Cloud).

The capacity — the cores and memory of the search hosts, the nodes that ran a meaningful share of the search processes (a utility node running its own few searches is listed but kept out of the basis) — is refreshed at most hourly by the Resource usage tracker and drives the automatic budgets of the resource policies. With an activity scope, it counts the search heads in scope and the indexers that serve them. A failed refresh keeps the previous record.

Policies run

Every 15 minutes the policies job:

1. reads the present inventory rows of every tier under the inventory lock (a long inventory publication makes it wait up to 90 seconds, then fail with inventory_publication_in_progress and retry at its next schedule) and evaluates the inventory policies with the completeness of each tier’s catalogs;

2. aggregates the indexed audit and resource records of the Activity policy window (15 minutes) and evaluates the activity and resource policies per account. The window ends where the collected evidence ends — the audit activity cursor, and the resource cursor when a resource policy is enabled — never at the current time, and never starts before the collected history. Consecutive windows always touch (with less than a whole window of new evidence, the window reaches back into what was already evaluated), so no minute is skipped; a run evaluates a further window only for a whole window of new evidence. Every aggregation is read whole; one Splunk reports as partial (missing peers, truncated output) is refused, and its window is evaluated again;

3. reconciles the findings and the Finding groups under the findings lock (see Findings and Finding groups), expires temporary dismissals, publishes the counts the card reads and writes one trackme:uam:finding event per transition;

4. delivers the operator notifications and the user notifications, and runs the automation rules (fail-open: a delivery failure never fails the run).

The receipt stores one evaluation per policy (enabled, state, evaluated / matched / excluded / unknown / not applicable counts, readiness, thresholds source) and its bounded insight (see Policy insight and preview). Completeness is per family: inventory findings are marked not observed only when every tier’s catalogs are complete, audit findings when the activity window was complete, resource findings when the resource coverage is complete on top. A rule whose aggregation failed is incomplete and neither creates nor clears anything. There is no findings limit: every match opens a finding. A very large live investigation queue (more than 50,000 findings one run can change) raises the Collection health warning findings_working_set_large; nothing is refused.

Bank holidays. A UAM tenant takes one effect of the bank holiday calendars, Suppress new alerts (Hold new findings). While a holiday of the tenant’s calendars applies and the effect is on, the policies set to hold (policies.<policy id>.hold_on_bank_holidays, defaults in The built-in policy catalogue) open no new finding:

• The run reads the holidays once, only when an enabled policy holds: the holidays that overlap the evaluated activity window, then the tenant’s calendars and effect. The outcome is holding, off (a holiday applies but the tenant does not hold) or unavailable (the calendars could not be read: nothing is held, a warning is logged).

• An inventory policy holds while a holiday is active at the time of the run (it judges the configuration as it is now). An activity or resource policy holds when a holiday overlaps its activity window: a burst on the last evening of a holiday is a holiday burst, even when the run comes after midnight.

• A match of a held policy whose finding does not exist, or is not observed (resolved, dismissed, or not seen by the previous run), is skipped before the capacity check: no finding, no Finding group, no transition, no trackme:uam:finding event, no operator notification, no findings action, and no findings capacity used. The direct user notifications leave its state untouched: a held match is neither a detection nor a step towards a recovery.

• A finding already observed is evaluated as on any other day (updated, escalated, no longer observed); expiries and housekeeping are unchanged. A held match never makes the run incomplete, and the first run after the holiday opens what was held.

When the outcome is not “no holiday”, the receipt carries bank_holidays: state, names (up to five holidays), held (the matches held) and policies (the enabled policies set to hold). Each evaluation that held carries its own held count, and the findings counters carry held while the run holds. The policy_evaluation endpoint returns the run’s bank_holidays.

Historical backfill

The backfill report collects the audit and resource evidence of the configured range before the live activity start (Backfill days: none — the wizard’s preset —, 7, 14, 30, 60 or 90 days in the wizard, 0–90 in Settings), through the route account. Evidence in an index cannot be taken back, so the backfill never collects a span twice, and nothing is skipped:

• The plan ends exactly where the live activity started and waits until the first audit activity and resource usage runs have set that point (backfill_waits_for_first_activity_run, backfill_waits_for_first_resources_run, normal receipts).

• Fixed buckets (Backfill bucket, 5 minutes by default) are walked backwards, newest history first, one audit read and one resource read each, with the live collection’s reads: a busy bucket is split, down to a few seconds read whole.

• A failing bucket is deferred, never skipped. Before any bucket succeeded in a run, a failure stops the run (a source that is down costs one attempt per hour, and the run is reported failed). Once one succeeded, the deferred buckets get their next attempt and the run goes on past a failure; three failures in a row stop it (backfill_source_failing). A bucket that keeps failing is halved to isolate the failing span.

• A bucket cut by the end of its run resumes at the next run after the last line it wrote; a run stops at Backfill max seconds per run (3,570 s). A bucket read while some peers did not answer is listed to be read again, like a live range.

• Retention edges come from the sources: the plan stops at the oldest _audit and _introspection data (retention_reached). The introspection retention is usually shorter (about two weeks), so the resource history may end earlier than the audit history.

• Re-plans carry the coverage forward: a changed range or bucket size, or Plan again, starts a new plan where the previous one stopped; a range already covered completes at once. A complete or cancelled plan un-schedules the report; backfill_days = 0 disables it.

• The backfill also scans each bucket for Splunk Web logins and automation evidence, so the people and services already active when the tenant was created are classified without waiting. It does not record CSV writer hints, and the inventory is never backfilled.

• Controls in Collection health → Trackers, on the expanded Backfill row: Pause (after the current bucket, progress kept), Resume, Cancel (retires the report; collected buckets stay indexed), Plan again, and Run now. The receipt carries the buckets done / split / retried / deferred / resumed, the events written, the progress (percent, estimated end from the measured rate) and the plan status. An existing tenant keeps its bucket size: set Backfill bucket (s) to 300 to move it to 5-minute buckets; the plan is rebuilt and the coverage already collected carried forward.

Receipts and diagnostics

A receipt is value-free: counts, statuses, diagnostic codes and window bounds — never an account name, a search text or a credential. Collection health renders the latest receipt of every job and the last seven days of runs from the indexed receipts, each diagnostic explained. A completed run with zero observations is a successful run: absence of activity is not a failure. A completed run is not a lossless-ingestion guarantee — the receipt says what was covered.

The evidence

Evidence events are written to the dedicated trackme_uam_events.log under $SPLUNK_HOME/var/log/splunk (UAM writes one line per search, so it never shares the state-events file of the other components) and routed by TrackMe’s ingest evaluation to the target index of their family.

Sourcetype	Family	One record per	Main content
trackme:uam:observation	evidence	Search and window	The account and app, the search id, type and status, the run metrics, the time bounds (the N/A all-time marker included) and the grant.
trackme:uam:resource	evidence	Search and window	The search’s resource usage: CPU seconds (search head and indexers) with their source, peak memory, elapsed time, nodes and samples.
trackme:uam:inventory	evidence	Inventory change	The object’s fields, the change (added, changed, removed, reappeared, reclassified) and the fields that changed.
trackme:uam:data_usage	evidence	Index reference and batch	One fact per joint key (tier, account, app, origin, path, index, sourcetype, read or write, named or unresolved) and audit batch, with its counters and up to five sample search ids. See Data usage.
trackme:uam:search_text	search text	Reported search	The SPL, written only while Store SPL search is on.
trackme:uam:finding	findings	Finding transition	The finding with its evidence and investigation state.
trackme:uam:receipt	receipts	Tracker run	The outcome of the run.

• Observation fields. account and account_source, app, sid, search_type, category, outcome, search_status, metrics (total_run_time, event_count, result_count, scan_count, exec_time), time_bounds (search_et / search_lt as text) and granted_time; use _time for the event time. The same sourcetype also carries one rest_access summary per account and batch (REST job access counted per kind: job_poll, job_lookup, job_dispatch, stream, remote, other) and the denied REST: lines (security evidence).

• Correlated executions are not indexed: the Activity tab, the account views, the REST API (activity with kind=execution) and the AI Agent correlate each search from its audit records when they read it (start and completion observed, ambiguity, the accounts, apps and search types reported). The trackme:uam:execution events of earlier versions stay searchable until they age out, and are read too.

• Resource fields. account, sid, app, search_type, provenance, mode, label, cpu_seconds, peer_cpu_seconds, cpu_source, peak_mem_mb, elapsed, nodes, samples, roles, window.

• Inventory fields. Additions, reappearances and removals carry the whole object; a changed or reclassified event carries only the changed fields, new (object.<field>) and earlier (previous.<field>), plus the owner.

• Data usage facts carry key (the JSON list of their dimensions) and the counters searches, index_searches, index_completed, late_searches and matched_events, written as the indexed fields uam_data_key, uam_data_refs, uam_data_n, uam_data_ndone, uam_data_late and uam_data_events, plus uam_data_index, uam_data_sourcetype and uam_data_label (the saved search that ran). A map over 30 days is one tstats over these fields, never a read of raw events.

• Finding transitions carry the evidence only when the policies run changes it (new, updated, reobserved); every decision field stays.

• Every event carries tenant_id (an indexed field) and environment_id; the audit and resource records also carry the indexed dimensions uam_kind, uam_environment, uam_account, uam_sid, uam_app and uam_status, so the per-account and timeline reads run through tstats. These index-time settings take effect after the splunkd restart of the app installation, on the instances that parse the UAM log; until then, and for older evidence, the readers fall back to slower raw searches.

• No field is written that nothing reads, while the fields with investigative value stay.

• Default indexes: the tenant’s summary index for evidence, search text and receipts, its notable index for findings. Each family can be pointed at another index in Settings → Indexes.

• One compact record per search and window: the grant folds into the completion (granted_time), REST job access is summarised per account rather than written line by line, null keys are dropped.

• Searches, dashboards, alerts or custom policies of your own that read sourcetype=trackme:uam:execution return nothing for activity collected since 2.4.21: rewrite them on sourcetype=trackme:uam:observation (one record per search and window, with sid, outcome, granted_time, category, search_status and the account fields).

Store SPL search and Fetch SPL

• Store SPL search (store_search_text) is off by default. When on, the SPL of every reported search is indexed as its own trackme:uam:search_text record in the search text index. Audit search text can carry literals and secrets: point the search text index at an index restricted to the people allowed to read it before you switch this on.

• Fetch SPL (account view, Activity tab, saved search view) shows the SPL of one search. It reads the stored trackme:uam:search_text record (or older evidence that carried the SPL inline) as the calling user; otherwise it reads the environment’s _audit record live through the route account, as the calling user, and stores nothing. When the text cannot be shown, a reason says why: not_in_audit, account_access_denied (your roles are not granted the route’s remote account), access_denied (the environment refused you), account_resolution_failed, authentication_failed, transport_error, search_incomplete, deadline_exhausted or audit_lookup_failed.

The saved search view

Opened from a saved search finding, the Scheduled searches tab or an account’s owned searches, the saved search view shows the inventory definition of the saved search on its tier, its runs (CPU, run time, peak memory, a CPU chart and the 50 latest runs), its findings and its change history. Fetch definition reads the current definition live from the saved search’s own tier, as the calling user (reasons such as not_found, account_access_denied, access_denied or tier_not_configured explain a refusal); nothing is stored. Runs are matched on name, app and owner (a nobody-owned saved search runs as splunk-system-user); a run of the same saved search by another account is left out.

Licence impact

The evidence is indexed data and counts against the Splunk licence like any TrackMe summary event. Its volume follows the search activity of the monitored environment — one compact record per search and window, one resource record per search and window, the inventory changes, one receipt per run. No row count caps it: UAM collects everything in scope. To reduce it, restrict the Activity scope to the search heads you need to monitor, switch off Collect search resource usage (the resource policies then stay learning), keep Store SPL search off, and size Backfill days with the retention of your sources in mind.

Data usage

The Data usage tab (Data usage) answers which indexes are read, by whom and through which path. This section describes the collection behind it and the rules its views follow.

The collection. The trackme_uam_data_usage_tracker_tenant_<tid> report (4-59/15 * * * *) has its own checkpoint in the audit trail, its own budget (870 s by default), lock, receipt and Collection health entry. It reads the audit trail through the same bounded, resumable reader as Audit activity, and writes no activity records. What a run cannot finish in its time continues at the next run; a batch whose extraction or writes failed is held and read again, never acknowledged.

• Resolution. Each search is parsed for the indexes and sourcetypes it names: literal index= and sourcetype= terms, IN lists, macros (with their arguments), event types, data models (their root constraints) and saved searches (savedsearch / loadjob), resolved through the published catalogs of the search’s own tier, with Splunk’s private / shared precedence. Wildcards stay patterns. A negated term never makes a read. A late field filter (after the first command) is a reference, not a source read. A collect destination is a write reference. So is the target index of a saved search’s summary indexing or Log event alert action, read from the published saved search catalog (inherited alert_actions.conf defaults included; an empty target means summary or main) and recorded on each run on the path summary_index or log_event. Custom alert actions are never inferred.

• Unresolved scope. What cannot be resolved is counted, not dropped: the search is recorded as scope unresolved with its reasons (a macro, event type, saved search or data model the catalog does not hold, a dynamic term, a wildcard role default, a truncated SPL, a resolution limit reached). For a search naming no index, the indexes its account’s roles give it (the role search filter’s indexes, else the default indexes the roles allow) are shown as context, never as reads.

• Counted once. A search is counted at its grant in the legacy Audit:[…] key=value format, or at its completion in the structured format. A canceled or failed completion counts; a search Splunk refused at its quota (action=quota info=denied) is not a read.

• The indexers’ confirmations. On a distributed deployment, the search peers record in remote_searches.log the search they received after Splunk expanded its macros and saved searches. The Data usage run reads those lines from its own cursor and keeps them as a separate facet (Indexers received), never added to the search head’s counts. A standalone search head has none.

• Declarations. Each catalog publication projects which knowledge objects name each index (saved searches, macros, event types, dashboards, data models). A declaration is not a read.

• History. When the tenant has a backfill, the Data usage history starts at the backfill depth, upgraded tenants included, but never before the oldest event of the audit trail: what _audit no longer holds cannot be read. Without a backfill, a new tenant’s history starts where its Audit activity started, and an upgraded tenant’s at its first Data usage run.

The rules of the views. Every case, with diagrams and worked examples, is in the Data usage reference: how each claim is processed.

• One indexing layer. Usage classes and review candidates are decided across every tier of the tenant: an index read on any tier is never a candidate. The indexes TrackMe writes to — the global index settings, every tenant’s overrides and this tenant’s own families — are never candidates.

• No absence from too little evidence. Until the collection covers the selected period, or seven days of it, no index is called unread and no knowledge object not run: the views say so far and give the date from which the claim can be made. A period with no Data usage run, an incomplete fact read or facts without the rollup key withhold every absence verdict. The first runs of a new tenant, before its first catalog listing, count as progress.

• Excluded apps (trackme and Splunk_SA_CIM by default) are shown, never counted: an index only they read is Excluded apps only.

• Wildcards reach by index name. index=cribl_* is matched against the indexes known for the tier with Splunk’s index pattern rule (* never matches an internal index), then narrowed to the indexes its reader’s roles allow; it is never narrowed by sourcetype. A wildcard reference never makes an index read by name.

• Role search restrictions. The roles of each reader are read from the published users and roles catalogs when the view is read: allowed indexes, minus the disallowed ones, narrowed by the index terms of the role search filters. A named index the roles do not allow is counted as denied by roles, never as a read. The scope is an upper bound, and the current roles apply to the whole period.

• Not due. A scheduled saved search that did not run, whose schedule leaves a longer gap than the period covers, is Not due in this period, never Not run.

• Least accessed ranks the indexes read by name with a daily volume above zero by reads per GB/day, and marks the lowest quarter. It needs at least four such indexes.

Volume and cost. A tier is linked to one VOL tenant (index volume) and up to 200 DSM tenants (incoming data), matched automatically whatever account they collect through unless set in Settings → Data usage. The cost per day comes from the VOL tenant’s costs (its Global license usage configuration), when they are enabled; otherwise no cost is shown, never a guessed one. A link grants nothing: every read repeats the caller’s visibility of the linked tenants. An index without a linked volume is counted, not weighed. Each index is weighed by its 30-day average of complete days; while VOL has no average for an index yet (its first day without a backfill), by its last 24 hours, and the views say so.

Reads without a time budget. The map, an index’s detail, the knowledge objects and the access timelines run as the caller’s own search jobs: the start request dispatches them and returns at once, the UI polls them until every one is done (naming what is still being read), and cancels them when the selection changes or the view closes. Only the 600-second safety timeout applies; a read that reaches it is reported incomplete, never shown as empty data. The start answer fixes the period, and every poll reads the same one. The linked VOL and DSM reads page through every record, each request bounded by splunkd_timeout.

Limits. At most 50,000 fact keys and 10,000 catalog indexes per tier are read per map; a larger answer is flagged truncated and withholds every absence verdict. An access timeline has at most 120 clock-aligned buckets and 10,000 aggregated rows, ten lanes per page. Every limit reached is said in the view.

Identity and classification

The account. Every observation carries account and account_source (legacy_search_user, structured_actor or unknown). The legacy user header wins when present: it names the caller, while the structured actor is often the system principal. It is a labelled source claim, not a verified initiator: UAM never claims that a person sat at the keyboard, and a scheduled search runs under its owner or the system account.

Classes and detection.

Class	Detected when	Limits
human	An interactive search — a resource record whose provenance starts with UI: (UI:Search, UI:Dashboard:<name>, UI:Report) and that carries no saved-search label — or a successful browser login to Splunk Web. One is enough; the detection is sticky and never demotes. Switch: Detection (human).	Ad hoc alone is not a signal (connectors dispatch ad hoc searches through the API); a labelled dispatch is not either (a dashboard set to run as its owner). Users working only through React apps (TrackMe’s UI included) reach splunkd as rest:jobs — the login signal covers them.
service	Sustained automation and nothing a person does: automated searches (REST, scheduler, acceleration, internal) on at least seven distinct days, never a Splunk Web search, a splunkjs page, an MCP client or any other provenance a person might use, and not a detected human. Withdrawn as soon as such a search appears. Switch: Detection (service).	The automation days come from the legacy key=value _audit format (provenance= / savedsearch_name=): a JSON-only _audit gives no service detection, and the seven days must lie inside the audit retention. An account with no activity is never detected.
system	Seeded for nobody, splunk-system-user and the built-in admin whenever unset; never detected, never demoted by a detection.	Assign it by hand to the tenant owner and to the administrator accounts that run the platform’s own workload.

Rules. Up to 100 ordered classification rules, each with a name, an enabled flag, a match type (exact or wildcard — *, ?, [abc], [!abc] on the whole name; no regular expression), a pattern, case sensitivity and the class. No naming heuristic runs unless an administrator saves a rule. Manual classes (up to 10,000, from Settings, the Users tab or its bulk classification) always win.

Precedence, in every reader (Users tab, account view, inventory, policies): manual class → system default → first enabled matching rule → human detection → service detection. Unmatched, undetected accounts stay unclassified. Removing a manual class returns the account to the automatic resolution; disabling a rule stops it at once; switching a detection off ignores its records without deleting them. Settings → Account classification previews the draft (up to 1,000 names, the effective class and its source) before saving; the Users tab shows a detected marker with the evidence next to the class. Changing a class never rewrites historical evidence nor resolves an investigation: the policies use it at their next run.

Privileged users

The Privileged users tab and Settings → Privileged users (privileged in the configuration) answer which accounts hold high privileges on the monitored tiers, and why.

Definition. An account is privileged when one of its roles, or a role that role imports (imported_roles, resolved transitively; a cycle is tolerated), is a privileged role or grants a privileged capability.

Inheritance. Splunk grants roles and capabilities through direct membership and through inheritance, and a role inherits from roles that inherit further: the assessment resolves the import chain breadth first. An inherited reason keeps its chain (path: the role held first, the privileged role last), an inherited capability names the role that grants it itself (granted_by, the nearest one; none when only Splunk’s imported_capabilities knows it), and each role keeps its own definition (its own capabilities and the roles it imports itself) apart from what it inherits. The tab words them: team_lead imports admin, team_lead inherits admin through ops, leaf inherits list_settings from base through mid.

Rule (key)	Default	Behaviour
Role name patterns (role_patterns)	*admin*	Live, case-insensitive globs (*, ?): a role created later is privileged as soon as its name matches.
Roles added (roles_include)	none	A role no pattern matches, made privileged by name.
Roles excluded (roles_exclude)	trackme_admin	Overrides a pattern. It removes only the name rule: a privileged capability the role grants still counts.
Privileged capabilities (capabilities)	admin_all_objects, change_authentication, delete_by_keyword, edit_roles, edit_tokens_all, edit_user, list_settings, list_storage_passwords, rest_apps_management	Granted directly (capabilities) or through imported roles (imported_capabilities and the capabilities of every role imported).

Up to 500 entries per list. A role name holds letters, digits and _ . -; a pattern also * and ?; a capability also :; a role cannot be both added and excluded.

Data. No new collection and no call to Splunk: the assessment reads the published users and roles catalogs of every tier (each role’s capabilities, imported_roles and imported_capabilities, each user’s roles), so it follows Users and roles every (60 minutes by default). An account present on several tiers is one account, and each of its reasons names the tier that grants it.

What the tab shows. Tiles for the privileged accounts (and their share of all accounts), the privileged roles, the privileged capabilities in use and the privileged accounts active in the last 24 hours; pies of why accounts are privileged (role, capability or both) and of their classes; a bar of the accounts per privileged capability; the privileged accounts with their reasons in words, last seen and executions over 24 hours; the roles of every tier with their rule, inherited roles and privileged capabilities. The activity is one tstats over the audit records, read as you, naming the privileged accounts (up to 1,000; above, it reads every account and keeps the privileged ones); when it cannot be read, the tab says the activity is unavailable, never that the accounts were inactive.

Settings. The patterns in a text area, then two dual lists filled from what the tiers hold: roles (moving a role a pattern matches out of the privileged list adds it to the exclusions; moving another role in adds it by name) and capabilities. The rules apply from the next read of the tab.

The tab describes; the four privileged policies (see Privileged policies) raise findings from the same assessment.

The built-in policy catalogue

A UAM tenant evaluates sixteen built-in policies in four families. Every policy has:

• Enabled — on or off.

• Severity — low, medium, high or critical (UAM has no informational level). A manual severity set on a Finding group survives later runs; choosing Members (the highest severity of its findings) restores the derived one.

• Excluded owners / accounts and Excluded apps — up to 500 exact, case-sensitive names each (no wildcard). Every policy except the CSV lookup footprint starts with the apps Splunk_SA_CIM and trackme excluded, so TrackMe’s own trackers and data-model acceleration do not flag the monitoring itself. Exclude owner from this policy in a finding menu adds the owner to this list.

• Account scope (account policies) — human (explicitly classified human), classified (human or service, never system) or all. Accounts outside the scope count as not applicable.

• Its thresholds, each compared strictly: a finding needs more than the limit.

• During bank holidays — Hold new findings or Keep opening findings (see Policies run). The six inventory policies hold by default: configuration reviews nobody handles on a day off, still there the next working day. The activity and resource policies keep opening findings by default: unusual activity on a day off is more telling, not less. A configuration stored before 2.4.19 reads each policy’s default. Changing it never resets the direct user notifications.

The policy editor (Policies → Edit) has four tabs: General settings (status, During bank holidays, severity, scope, thresholds, automatic budgets, exclusions, Preview policy), Email template (the direct user emails of this policy, see Notifications and communications), Actions and Automations (findings actions, see Findings actions).

Inventory: saved searches

Policy	What it detects	Default	Severity
personal_schedule/v1	An enabled schedule owned by an account classified human.	On	Low
unclassified_schedule/v1	An enabled schedule whose owner has no human or service class.	Off	Medium
dense_schedule/v1	More scheduled runs per active hour than the limit.	Off	Medium
all_time_schedule/v1	A schedule with no lower time bound (all time).	On	Medium
realtime_schedule/v1	A schedule with a real-time dispatch range.	On	Medium

Inventory: accounts

Policy	What it detects	Default	Severity
csv_lookup_footprint/v1	A CSV lookup file too large, or an account’s CSV lookups too large in total.	On	Medium

Activity: accounts

Policy	What it detects	Default	Severity
launch_burst/v1	An account launching an unusual number of searches.	Off	Medium
costly_search/v1	An account running repeated long searches.	Off	Medium
all_time_search/v1	An account running all-time searches.	On	High

Resource: accounts

Policy	What it detects	Default	Severity
resource_consumption/v1	An account’s search CPU above its allowance.	On	Medium → High
resource_intensity/v1	An account running repeated searches heavy in CPU or memory.	On	Medium
resource_deviation/v1	An account’s CPU today unusual for its daily baseline.	Off	Medium

Privileged: accounts and roles

Policy	What it detects	Default	Severity
privileged_account_new/v1	An account that became privileged (reported once per change).	Off	High
privileged_role_capability/v1	A role whose own definition now grants more privileges (reported once, about the role).	Off	High
privileged_account_dormant/v1	A privileged account without activity for the period.	Off	Medium
privileged_reference/v1	A privileged account the reference does not declare, or a stale reference.	Off	High (stale: Low)

Default settings

What each policy applies to, its account scope, what it does on a bank holiday, and its thresholds with their defaults.

Policy	Applies to	Bank holidays	Thresholds and notes (defaults)
personal_schedule/v1	Saved search	Hold	No threshold. Common where administrators own every schedule, hence low.
unclassified_schedule/v1	Saved search	Hold	No threshold. A governance gap, not evidence of misuse.
dense_schedule/v1	Saved search	Hold	Maximum scheduled runs per active hour 12 (1 to 60). Configured frequency, not observed launches.
all_time_schedule/v1	Saved search	Hold	No threshold. dispatch.earliest_time of 0 or empty.
realtime_schedule/v1	Saved search	Hold	No threshold. An explicit rt bound.
csv_lookup_footprint/v1	Account, scope all	Hold	100 MiB per file, 500 MiB per account and tier. No app excluded by default.
launch_burst/v1	Account, scope human	Keep opening	Maximum distinct searches per window 200.
costly_search/v1	Account, scope human	Keep opening	Minimum run time 300 s, maximum long searches 3.
all_time_search/v1	Account, scope human	Keep opening	Maximum all-time searches 0.
resource_consumption/v1	Account, scope classified	Keep opening	Allowance 10 % of the search CPU capacity (else 3,600 s). Raised one severity level above a dominant share of 40 %.
resource_intensity/v1	Account, scope classified	Keep opening	A heavy search uses 2 % of the search CPU or 10 % of the smallest search host memory (else 300 s or 2,048 MB); maximum heavy searches 3.
resource_deviation/v1	Account, scope classified	Keep opening	3 × the daily p95 over 14 days, floor 3,600 s. Off, in preview mode: matches are recorded, no finding and no notification.
privileged_account_new/v1	Account, scope all	Keep opening	No threshold. Splunk-operated accounts left out (Include accounts operated by Splunk, off).
privileged_role_capability/v1	Role	Keep opening	No threshold. Exclusions name roles.
privileged_account_dormant/v1	Account, scope all	Hold	Days without activity 30 (7 to 365). Splunk-operated accounts left out.
privileged_reference/v1	Account, scope all	Keep opening	No declared account and no lookup table (the policy stays learning), Report stale references on. Splunk-operated accounts left out.

The server supplies the titles, descriptions, bounds and remediation of every policy; the sections below give the full detail.

Inventory policies — saved searches

The subject is one saved search of one tier; the finding keeps the same identity across runs. Explicitly unscheduled or disabled searches are not applicable; a missing flag or an unreadable bound is unknown, never assumed. Evidence: the owner and its class (with where the class comes from), the app, the tier, the schedule and the bounds read. UAM never edits nor dispatches the observed searches — the remediation is advisory. These policies follow the catalog cadence: the saved-search listing is daily by default.

Policy	Details
personal_schedule/v1 Scheduled search owned by a human account	Counts: enabled schedules whose owner is explicitly classified human (unknown and service owners do not match). Thresholds: none. Common where administrators own every schedule, hence the low severity. Suggested action: review whether this schedule should run under an approved service account, or record an owner exception.
unclassified_schedule/v1 Scheduled search owner is not classified	Counts: enabled schedules whose owner has no human or service class. A governance gap, not evidence of misuse. Thresholds: none. Suggested action: classify the owner in Settings after confirming the account’s purpose.
dense_schedule/v1 Scheduled search exceeds the hourly schedule limit	Counts: distinct configured minute slots in an active clock hour of a standard five-field cron (wildcards, steps, ranges and lists; duplicates count once; named days, macros and extended forms are unknown). Configured frequency, not observed launches. Thresholds: maximum scheduled runs per active hour, 1 to 60 (default 12). Suggested action: review the business need and reduce the frequency, or add a justified exception.
all_time_schedule/v1 Scheduled search has an all-time dispatch lower bound	Counts: enabled schedules whose dispatch.earliest_time is 0 or empty (Splunk’s default: no lower bound). Inline SPL time modifiers may still restrict the search. Thresholds: none. Suggested action: configure a bounded dispatch time range where appropriate.
realtime_schedule/v1 Scheduled search uses a real-time dispatch range	Counts: enabled schedules with an explicit real-time bound (rt, rt-5m@m, rt@w0…). An unsupported rt form is unknown, never historical. Thresholds: none. Suggested action: review whether a scheduled historical search can meet the requirement.

CSV lookup footprint

csv_lookup_footprint/v1 — CSV lookups attributed to an account exceed its size allowance — is an inventory policy whose subject is an account:

• What it detects, separately on every search head tier: one CSV lookup file larger than Maximum individual CSV file size (default 100 MiB), or the cumulative size of the CSV lookups attributed to one account larger than Maximum cumulative CSV size per attributed account and tier (default 500 MiB). Both 1–1,048,576 MiB.

• On by default, severity medium, account scope all, no app excluded by default.

• Attribution: probable writer, recorded owner as fallback. The Audit activity tracker records, from the audit evidence it already reads, which account last wrote each lookup file. A file is attributed to that probable writer when the evidence supports it, else to its recorded owner (a shared file owned by nobody falls back to its recorded owner unless audit evidence names a writer). The evidence keeps the basis of the attribution and both identities. Attribution is best effort since monitoring began, not proof of authorship: missing audit evidence, ambiguous targets, concurrent writes or later unmatched changes use the owner fallback, and the backfill does not provide writer hints.

• Size is the stored file size (a compressed CSV counts at its stored size), not the replicated bundle size. Only the metadata is read — never the contents — and no third-party app is required.

• Only tiers whose lookup-file catalog is complete and less than an hour old are evaluated; a missing, partial or stale catalog can never clear a finding.

• Suggested action: review the largest files and their generating searches with the probable writer or the owner; reduce retained rows or columns, use the KV store where appropriate, review bundle replication settings with an administrator. UAM never deletes a file nor changes replication. Direct user emails of this policy list up to 20 of the largest attributed lookups.

Activity policies — accounts

Aggregated from the trackme:uam:observation records of the evaluated Activity policy window (15 minutes of collected evidence by default, see Policies run), one row per search first, then per account and app. The counts are distinct search ids the Audit activity tracker observed — a launch-count proxy — labelled with the coverage (complete / partial). One finding per policy, environment and account: a repeated burst updates the same finding. Evidence: the window, the count, the thresholds, the account class and scope, the top five apps and, for costly_search/v1, the maximum and total run time. Excluded apps are removed from the account’s aggregate before the threshold is compared.

Policy	Details
launch_burst/v1 Account launched an unusual number of searches	Counts: distinct searches by one account in the window. Thresholds: maximum distinct searches per window, 1 to 100,000 (default 200). Suggested action: confirm whether the burst is expected automation; raise the limit or record an account exception if it is.
costly_search/v1 Account ran repeated long searches	Counts: distinct searches whose reported run time reached the minimum. Run time is the audit completion figure; CPU is not estimated. Thresholds: minimum run time 1 to 86,400 s (default 300); maximum long searches 0 to 100,000 (default 3). Suggested action: review the long searches (run time, scan count); work on time ranges, scheduling or acceleration.
all_time_search/v1 Account ran all-time searches	Counts: distinct completed searches whose reported earliest bound is explicitly unbounded (legacy search_et=N/A, structured 0). A missing bound is unknown. Thresholds: maximum all-time searches, 0 to 100,000 (default 0). Suggested action: agree on bounded time ranges, or an account exception for approved investigations.

Resource policies — accounts

Read from the trackme:uam:resource records (search heads and indexers, ad hoc and scheduled), through one bounded rollup per policies run: CPU adds up across windows, memory keeps its peak. The system class is outside the default classified scope (data-model acceleration never becomes a finding by default) but its CPU stays in the share denominator: it is real load.

Automatic budgets. With automatic thresholds on (the default), the absolute limits are derived from the capacity — the core sum of the search hosts and the memory of the smallest one — and clamped to each threshold’s own range:

Policy	Percentage (default)	Derives
resource_consumption/v1	Allowance as a share of the search CPU capacity (10 %)	Maximum CPU seconds = search cores × window seconds × percentage
resource_intensity/v1	Heavy search CPU as a share of the search CPU capacity (2 %)	Minimum CPU seconds of a heavy search = search cores × window seconds × percentage
resource_intensity/v1	Heavy search memory as a share of the smallest search host (10 %)	Minimum peak memory of a heavy search = smallest search host memory × percentage

With automatic thresholds off, the stored absolute values apply. With them on but no usable capacity, the stored values apply and the evaluation says manual_fallback. The evidence carries the thresholds source (auto / manual / manual_fallback) and the capacity basis; the policy editor shows the effective values.

Learning period. The resource rules are learning — nothing evaluated, no finding created, nothing marked not observed — until the tenant holds Resource rules: minimum history (24 hours by default) of resource history (the backfill counts) and, for automatic budgets, a known capacity. The Policies tab shows the reason.

Policy	Details
resource_consumption/v1 Account’s search resource consumption exceeded its allowance	Counts: the CPU seconds of the account’s searches in the window, against its allowance. Its share of the environment’s search CPU is always in the evidence; strictly above the dominant share the severity is raised one level. The share never opens a finding on its own. Thresholds: maximum CPU seconds 1 to 10,000,000 (default 3,600, derived when automatic); allowance 1 to 100 % (default 10); dominant share 1 to 100 % (default 40). Suggested action: open the account’s most expensive searches; work with the owner on time ranges, acceleration, scheduling or workload rules; raise the allowance or record an exception for approved workloads.
resource_intensity/v1 Account ran repeated resource-intensive searches	Counts: distinct searches whose CPU seconds or peak memory reached the per-search minimum (a search spanning several windows is summed; memory keeps its peak). Thresholds: minimum CPU seconds of a heavy search 1 to 1,000,000 (default 300, derived); minimum peak memory 64 to 1,000,000 MB (default 2,048, derived); maximum heavy searches 0 to 100,000 (default 3); automatic percentages 2 % CPU and 10 % memory. Suggested action: review the heavy searches (type, provenance, label); agree on bounded time ranges, acceleration or scheduling.
resource_deviation/v1 Account’s CPU consumption is unusual for its daily baseline	Counts: today’s accumulated CPU (UTC day) against max(multiplier × historical daily p95, floor), after at least the minimum number of fully covered prior days since the account was first seen. Missing coverage is unknown, never zero; today is never extrapolated. Thresholds: historical lookback 7 to 30 days (default 14); minimum complete days 7 to 30 (default 7, at most the lookback); multiplier 1 to 20 (default 3); minimum daily CPU seconds 1 to 100,000,000 (default 3,600). In preview mode by default: matches are recorded in the evaluation, no finding and no notification. Suggested action: review the daily baseline, the coverage and the expensive searches; confirm workload changes before adjusting the multiplier.

Privileged policies

The four privileged policies read the privileged assessment (see Privileged users) of every tier, from the published users and roles catalogs, in the policies run. They have no excluded apps (they judge roles and accounts, not searches) and never email the account a finding is about.

Changes, reported once. The two change policies compare the current assessment with the previous one, which the policies run keeps (a snapshot, per tier). A change is matched by the run that finds it and opened as a finding; the next complete run no longer matches it, so a resolve sticks and a later change opens a new episode. A change held by a bank holiday stays pending and is opened by the first run after it. The first evaluation, a change of the Privileged users rules, and a change policy turned on record the current state without reporting. A partial users or roles catalog never moves the snapshot, never matches and never clears.

Policy	Details
privileged_account_new/v1 Account became privileged	Matches: an account privileged now that was not at the previous assessment of its tier, whose own roles changed (roles_changed) or that is new (account_created). An account made privileged with its own roles unchanged is the role policy’s finding; with the role policy off, or the originating role excluded from it, it is this policy’s (privilege_inherited, naming the roles whose definition changed). Evidence: the roles before and after, the reasons with their chains, and, when the audit trail holds the change (create_user / edit_user), who made it and when. Suggested action: confirm the grant was approved; otherwise remove the privileged role and review the account’s activity since.
privileged_role_capability/v1 Role gained a privileged capability	Matches: a role whose own definition changed — its own capabilities (capabilities_gained), the roles it imports itself (imports_changed) — so that it grants more privileged capabilities or inherits a privileged role, or a new role created privileged (role_created). The roles inheriting it are not reported again: the finding lists them (descendant_roles_reached) with every account newly privileged through any of them. A role whose inheritance grew while no role in its chain changed its own definition in the catalog is reported on itself (inheritance_changed). Subject: the role, on its tier (a Finding group of its own); findings actions read its name as $subject.name$. Evidence: the capabilities gained and the role granting each, the own capabilities and imports before and after, and, from edit_role, who made it with the capabilities before and after. Suggested action: confirm the change was approved; otherwise restore the role’s capabilities or imports.
privileged_account_dormant/v1 Privileged account is dormant	Matches: a privileged account with no search in the audit evidence (one tstats over the period, naming the privileged accounts), no Splunk Web login or interactive search seen by the Account signals tracker, and no successful login Splunk reports on any tier (last_successful_login, collected by the users catalog without making it an inventory change), for Days without activity. Never a guess: an account in the inventory for less than the period is not applicable; while the tenant’s audit evidence (the live history, plus a completed historical backfill) does not cover the period, the account is unknown. Not seen: activity on a search head outside the Activity scope, and REST calls that run no search.
privileged_reference/v1 Privileged accounts differ from the reference	Reference: the account names typed in the policy (every tier) and each lookup table, read with | inputlookup <lookup> on its deployment — the local search head, or a remote deployment through its TrackMe remote account — from the TrackMe app: the account field, and the tier field when set (an empty tier declares every tier). Matching is case-insensitive; * matches any text in typed names and in a lookup set to Wildcard (Exact, the default, ignores a value holding *). Up to 20 lookups of 500,000 values each; TrackMe’s own lookups are refused. Matches: undeclared — privileged on a tier the reference does not declare it for; stale_reference (with Report stale references, at low severity) — an exact declared name holding no privilege, or absent from every tier (a pattern is never stale). The evidence names the sources that declare the account. Never a guess: a lookup that cannot be read, reads empty, is too large or is not read in time keeps its last good values (diagnostic privileged_reference_unreadable, note reference_lookup_cached); a lookup with no good values yet, or a reference that declares no account, judges nothing; without a source, the policy is learning. A tier field with no value in any row is treated as unreadable (a renamed column).

Accounts whose collected email address is support@splunk.com — the accounts Splunk operates on a Splunk Cloud stack — are left out of the account policies unless Include accounts operated by Splunk is on. How it is evaluated shows the records judged and, for the dormancy and reference policies and the change author read, the exact searches.

Policy insight and preview

• Policy insight — the Last evaluation and the count cells of the Policies tab open what the last run saw: a verdict, the population, why records were unknown, excluded or not applicable (owner_unclassified, missing_<field>, cron_unreadable, excluded_owner / excluded_app, unscheduled / disabled, out_of_scope_<class>…), the owners and apps behind each bucket (owners can be classified in place), up to 25 samples, the run (window, readiness, thresholds source, inventory per tier) and a seven-day history per policies run.

• Preview policy (General settings) evaluates the draft against the latest stored inventory of the tenant without writing anything and shows the totals and up to 20 matching examples. It needs a collected inventory.

Custom policies

Tenant administrators write their own policies in SPL (Policies → Create policy). A custom policy runs on its own schedule and its findings appear with the built-in ones, in the same Finding groups.

Definition.

• Title, description, default severity and remediation (shown on its findings).

• Source tier — a policy about accounts reads the route account (the indexing layer); a policy about saved searches reads the selected tier. App context — the search namespace.

• SPL — up to 64 KiB. Time range — relative, from -<n><s|m|h|d|w> to now or an earlier relative bound, at most 31 days in the past; no all-time, real-time, future bound or calendar snap. Schedule — a numeric five-field cron.

Map search results to findings.

• Subject type: account (the result field holding the account) or saved search (the fields holding the owner, the app and the name — three distinct fields; alias them in the SPL). The subject places the finding in the right Finding group.

• Dimensions (up to eight fields) — distinguish several findings about one subject.

• Evidence fields (up to eight), a count field (occurrences per row, else one per row) and a time field (epoch seconds).

• Advanced limits, for example the result rows read (10,000 by default, at most 100,000), the findings per run (1,000, at most 10,000), 25 findings per subject, the samples and evidence bytes kept per finding.

• Changing the subject mapping or the source requires a higher identity version before publication: a new identity, so earlier findings are not reused for a different meaning.

• Bank holidays — Hold new findings or Keep opening findings (hold_on_bank_holidays, off when absent). It is not part of the identity, so turning it on needs no new identity version; it takes effect when published. A custom policy holds while a holiday of the tenant’s calendars is active at its anchor, the end of its search window: a holiday that ended earlier inside a longer window no longer holds. Held results publish no finding and use no findings capacity; a finding already observed is still evaluated. The run’s commit records held and, when a holiday applied, bank_holidays (state and names), shown as Held N (bank holiday: <names>) on the run.

Lifecycle.

Step	What it does
Draft	Save keeps a saved version of the draft (the 30 most recent are kept). Saving never publishes nor enables.
Preview	Runs the current form — saved or not — as you, asynchronously and cancellable (60 seconds, 1,000 result rows), and shows completeness, diagnostics, counts and at most 20 candidate cards. Nothing is published; a zero-row result cannot verify a mapping.
Publish	Freezes an immutable published revision (up to 100 per policy). Draft edits never change what is published.
Enable / Disable	Schedules or unschedules the policy’s report trackme_uam_custom_<id>_tenant_<tid>, owned by the tenant owner.
Archive	Irreversible: the policy stops, its definition and findings stay readable and it can be duplicated.
Delete permanently	Removes the definition, its versions, its scheduled search and its execution history; findings, evidence and notes remain.
Version history	Load, compare, restore (as a new draft) or delete a version; the published revision is protected.

Limits. 100 definitions per tenant (archived ones included) and 25 enabled. A scheduled run has 300 seconds and 10,000 result rows; at most 1,000 findings are kept per custom policy and 10,000 per tenant, separately from the built-in findings limit. Four custom policy runs at a time per installation, two per tenant, one per policy (previews included).

Execution health. Collection health → Trackers, family Custom policies, shows, per policy, the requested and actual schedule, the cron and next run, the last complete evaluation, the failure count, the recent runs and any running search; states are disabled, never run, healthy, healthy (no match), incomplete, failed, cancelled, capacity deferred, schedule drift, missed schedule and recovery required. Run now runs the published revision (findings may be published) and Recover frees a run whose searches are verified stopped. The Configuration Guardian reports the problems (see Configuration Guardian).

What differs from built-in policies.

• Custom findings are not notified: the operator and user notification policy selectors list built-in policies only.

• There is no Exclude owner from this policy action: write exclusions in the SPL.

• A custom policy can only map to accounts and saved searches.

Warning

The SPL of a custom policy is arbitrary SPL run on a schedule as the tenant owner (with the nobody owner, as Splunk’s internal scheduler identity), and a preview runs it as its author: a command or macro with side effects has them, preview included. Only trusted tenant administrators can author policies; give remote accounts least-privilege credentials.

Findings and Finding groups

The shared lifecycle, triage and grouping concepts are explained in A finding and its lifecycle and Finding groups; this section gives the UAM specifics.

The finding

A finding is one row per policy, environment and subject (a saved search, an account, or a role for the role policy), with a stable id, its status (open, investigating, resolved, dismissed), its severity (policy or manual), its assignee, a short investigation note, observed, first / last seen, episodes, reopened_count, the dismissal deadline, a bounded history (the 50 most recent records) and the evidence.

Transition	Meaning
new	First match: the finding opens.
updated	Severity or evidence changed while observed (a threshold edit updates the evidence without changing the identity).
reobserved	Matched again after an absence: a new episode; open and investigating findings only gain the history record.
unobserved	Not matched by a complete evaluation: the finding stays, its Condition reads not observed. It is never resolved for you; a partial evaluation never marks a finding not observed.
reopened	A resolved finding observed again returns to open.
expired	A temporary dismissal ended: the finding returns to open at the next policies run, observed or not.
triage	An operator decision; the policies never overwrite it.
retired	The policy no longer exists and its replacement evaluated fully: resolved by the system, not notified.

Decisions

Decisions are made on the Finding group (one group, or up to 500 groups at a time), fenced on the group’s version (a decision captured on an older version is refused, finding_group_version_changed):

• Status — Investigate, Resolve, Dismiss, Reopen. Resolve says the investigation concluded: the policy reopens the finding the next time it observes the condition. Dismiss says the finding is accepted: it stays dismissed while observed. Dismiss permanently (“until reopened by hand”) or for 1, 7, 30 or 90 days (API: one hour to 366 days); an ended dismissal returns the finding to open. Any status change clears a deadline; a new duration replaces it.

• Snooze notifications — 1 hour, 1 day, 7 days or 30 days (API: 60 seconds to 90 days): the operator emails about the group are held; the status does not change. Transitions during the snooze are not sent later.

• Severity — a manual severity; Members restores the severity derived from the group’s findings.

• Assign — to a user or an assignment group (see Assignment and automatic routing).

• Notes — the shared TrackMe Notes (Markdown, permanent or expiring) on the group or on a finding, separate from the short investigation note recorded in the history.

Every change leaves a history record with its author (an operator, the policy, the system or the AI Agent). Decisions on findings and groups (triage, group_members) require a TrackMe administrator of the tenant; a power user of the tenant can run findings actions, send an email action and set an assignment.

Finding groups

Automatic grouping. All findings about one account and the saved searches it owns form one Finding group — for example the two personal schedules of analyst-a and its all-time-search finding. Ownership defines the grouping; classification does not. A saved search without an owner keeps its own group. Every tier of the indexing layer shares the account’s group; different tenants never do. A role finding (the privileged role policy) keeps a group of its own. Each finding keeps its own subject, tier, policy, evidence and id. An automatic group holds up to 5,000 findings.

The group is the triage unit. A decision applies to every member except a member dismissed on its own (Dismiss only this finding): that member keeps its own dismissal until it ends or is reopened, and the group’s decisions leave it alone. A new kind of problem on a dismissed group reopens the group and keeps the earlier members dismissed on their own: a dismissal never silences a new problem. The group’s severity is the highest of its observed members unless an operator set one.

Manual grouping. Select groups on the Findings tab:

• Merge into one Finding group — 2 to 25 groups become one manual group (optional title). The most open status wins; a member whose decision differs keeps it on its own.

• Add to a manual group, Remove from the Finding group (up to 25 findings at a time) and Ungroup.

• An object’s findings always move together: removing a saved-search finding removes every finding of that object; removing an account returns it with its owned searches.

• A manual group holds up to 50 findings; later findings of a full manual group go back to their automatic group. Automatic grouping never undoes a manual grouping.

Assignment and findings actions

• Assignment (Settings → Assignment): up to 100 assignment groups (name, description, optional team email) and up to 100 ordered automatic assignment rules (by policy, severity, subject type, owner, app, environment), with optional emails to the assigned group. See Assignment and automatic routing.

• Findings actions (Settings → Finding actions): up to 50 administrator-defined drilldown searches, links, webhooks (through system-wide Webhook accounts) and the email action, run by hand from a finding or group menu, and up to 100 automation rules that run them for a policy (Policies → Edit → Automations). See Findings actions.

Finding events

Every finding transition is indexed as trackme:uam:finding in the findings index (the tenant’s notable index by default): build your own Splunk alerts, Enterprise Security correlation searches or SOAR playbooks on them.

Notifications and communications

UAM has three email channels, all independent of each other. The shared principles are in Notifications.

Notifications to operators

Settings → Notifications to operators (notifications), off by default:

• Events: New finding, Finding reopened, Severity raised, Finding resolved (on by default), Finding dismissed, Finding assigned. Being observed again or not observed never notifies.

• Minimum severity (medium), Policies (empty = all built-in policies), Delivery account (a TrackMe email account, see Configuration; empty = the search head’s local mail transport), Recipients (up to 50, filtered by the account’s allowed domains), Mode: Digest (one email per policies run, grouped by event then severity) or Per finding (one email per transition, threaded per finding in the mail client).

• Transitions are delivered at the end of the next policies run (within about 15 minutes); a failed send is retried at the next runs, and transitions older than 24 hours expire. A snoozed group’s transitions are not sent.

• Send test email sends a synthetic message with the saved settings, even when notifications are off.

• Every email links to the finding (TenantHome?tenant_id=<tid>&component=uam&tab=findings&finding=<id> opens its Investigate view).

Notifications to users

Settings → Notifications to users (user_notifications), off by default: one email to the account a finding is about, on detection and when the condition is no longer detected.

• Addresses come from Splunk’s user records, collected per tier by the users catalog and kept in a private store (never in the inventory, the evidence or the findings), or from an administrator’s Override email address (Users tab). A saved-search or CSV finding uses its own tier; an account finding needs one agreed address across the inventoried tiers, or the authoritative identity tier you choose. Missing, stale or conflicting addresses defer the email (at most 24 hours).

• Who receives: never a system account; human accounts by default; service and unclassified accounts only when you opt in.

• Detection and clearing: a new match opens an episode and sends one detection email. Two consecutive complete evaluations without the condition close the episode and send the “no longer detected” email (only after a sent detection, to the same recipient). Partial, unknown or learning evaluations never clear. A recurrence within repeat hours (24) does not send a new detection.

• Baseline: the first enabled evaluation records the current matches without sending them, unless Include current detections once is on.

• Independent of operator triage: resolving, dismissing, snoozing, grouping or assigning a finding does not stop or trigger user emails.

• Content: a common banner, introduction, footer, reply-to address and an HTTPS support link, plus per-policy subjects and bodies (Policies → Edit → Email template; every built-in policy is on except unclassified_schedule/v1; the privileged policies never email the account). Plain-text templates with the placeholders {account}, {policy}, {search_name}, {app}, {environment}, {summary}, {action} and {window}; subjects up to 256 characters, bodies up to 10,000.

• Limits: 50 emails per policies run (1–200), 3 per address per hour (1–20), 24 hours between repeat detections (1–168). One recipient per email.

• Preview and Send test email (to a test recipient, for one policy) use synthetic data and work while the channel is off.

Send email communication

Send email communication (finding and group menus, tenant administrators and power users) prepares a reviewed one-off email about a finding, a group member or a whole Finding group (up to 200 findings, one section per finding): the modal shows the findings, the recipient (which you may change for this email only), a full preview, and asks for explicit confirmation. It is sent once from a search process, never retried blindly: when the outcome is uncertain, check the inbox before sending again. It does not change the findings or the automatic notification state.

The UAM AI Agent

The UAM AI Agent reviews UAM findings with the evidence UAM collected; it replaces the UAM Findings Advisor of 2.4.18. Its modes, notes and decisions are those of every findings AI Agent: see Findings AI Agents.

• Inspect writes one current [AI Agent] note per finding (assessment, confidence, recommendation) and changes nothing. Act also applies the decisions the tenant allows.

• Ask the AI Agent (one finding, from its menu or the Investigate view), Run a review now (AI Agent tab) and the scheduled triage (hourly, off by default).

• The scheduled triage reviews at most findings per run findings, one per Finding group, skips a group reviewed within the review interval unless it changed, and never overrides an operator decision. Its selection uses the shared AI filters: by default only critical and high findings.

• The AI Agent tab shows the scheduled triage status and the decision log (kept 90 days).

• Requirements: AI enabled with a provider configured; for the scheduled triage, a tenant owner that is not nobody or splunk-system-user (Guardian ai_automation_owner_is_system_user). Launching, deciding and reviewing need a TrackMe administrator of the tenant; tenant power users can read the settings and the decision log.

Setting (vtenant_account)	Default	Meaning
ai_uam_agent_enabled	0	Scheduled triage on or off.
ai_uam_agent_mode	inspect	Mode of the scheduled triage (inspect or act).
ai_uam_agent_provider_name	empty	AI provider; empty = the first available.
ai_uam_agent_min_days_between_reviews	1	Days before a reviewed Finding group is reviewed again (unless it changed).
ai_uam_agent_max_findings_per_run	10	Findings reviewed per scheduled run, one per group (1–50).
ai_uam_agent_max_runtime_sec	3000	Time budget of one hourly run, in seconds.
ai_uam_agent_allow_investigating	1	Act may mark a group investigating.
ai_uam_agent_allow_dismiss_temporary	1	Act may dismiss for a time — never a critical or high finding still observed.
ai_uam_agent_dismiss_max_days	7	Longest dismissal, in days (1–90).
ai_uam_agent_allow_resolve	0	Act may resolve a finding its policy no longer observes.
ai_uam_agent_allow_classify_account	0	Act may set the class (human or service) of an account without an operator class.
ai_uam_agent_custom_instructions	empty	Plain-English instructions of the tenant administrators, followed by every run of the agent, asked or scheduled (at most 4,000 characters, line breaks kept).
ai_automated_priority_filter	critical,high	Shared: the finding severities the scheduled triage reviews.
ai_automated_filter_expression	empty	Shared: a filter expression on the finding (policy_id, owner, app, account, subject_type, status…).

These settings are edited in Settings → AI Agent of the workspace; Manage AI Agents shows the UAM AI Agent of each UAM tenant and edits its schedule, mode, provider, review interval, runtime and custom instructions. A permanent dismissal, an assignment, a severity change and any policy or configuration change are never available to the agent. See also The AI Advisors.

Custom instructions and Evaluate. The agent’s custom instructions (Settings → AI Agent → Custom instructions, also in Manage AI Agents automation and the Configure tenant dialog) come last in its prompt, after the provider’s custom prompt and, for scheduled runs, after the tenant’s shared custom instructions (ai_automated_custom_instructions, scheduled runs only). They never widen what the agent may do: the allowed decisions and the checks TrackMe makes before every decision still decide. Evaluate… runs the agent in inspect mode on the unsaved draft (empty = without instructions), on one open finding or a review of the open findings (at most 10, the most severe first), optionally side by side with a run without custom instructions. It writes nothing — no note, no decision, no review record — and is recorded with the status evaluation, so a later act run never starts from it. See AI Agents for findings.

Collection health and Configuration Guardian

Collection health

The Collection health tab is the first stop when something looks wrong. Everything that says something is wrong comes with its explanation, its evidence and its action, and every artefact UAM writes is named, located and one click away. Four views:

• Status — one row per tracker that needs attention, from the same assessment as the uam_collection_health Guardian check (so the page and the alert never disagree), plus the tracker’s last receipt. Expanded: every reason with its meaning and where it was reported (run, dataset, tier, window), What to do, the facts of the last run (status, finish, consecutive failures, search job, report and its live schedule, lock), the policies a policies run could not complete (Show why opens the policy insight), the evidence shortcuts (Receipts, Tracker log, Search job, Report, Scheduler log) and Run now / Recover. Below: Problems in the last 24 hours, the failed and incomplete runs of each tracker from the indexed receipts, recovered or not (the backfill only while a plan is active or a problem is current). Flow opens a read-only topology of the collection pipeline — the tiers and the indexing layer, the trackers, the KV collections and indexes they write, the policies, findings and their consumers — coloured by health.

• Trackers — every tracker in one table: status (opens the last-run view), live schedule (read from Splunk, not the defaults), last run, last collected, duration (with the budget applied, see Run budgets), diagnostics, actions. Filters: name, family, status (needs attention, failed, running, not scheduled), tier. Expanded: the activity checkpoint and the windows of the last run with their outcome (acknowledged, split, resumed next run, retried) for Audit activity; the resource windows, re-reads and capacity for Resource usage; the scans and their re-reads for Account signals; the datasets of a catalog; the backfill plan and its controls. Searches of the last run (Audit activity, Resource usage, Account signals, Policies) lists every search the run sent, exactly as sent — audit windows with the clause that resumes them, recounts, re-reads, resource windows, scan chunks, each policy aggregation per evaluated window — with its time range, Copy and Open in Search; only searches the run really sent are listed, with a note when the settings changed since or the run read through a remote account. Custom policies have their execution health under the Custom policies family.

• Data & storage — what UAM writes for the tenant: the events per index and sourcetype (volume over 1 hour to 7 days, latest event, search), the KV collections (records, size, lookup definition, a record viewer with a KV query, what large collections are made of), the logs and every scheduled report with its live state, owner and next run.

• Troubleshoot & help — the diagnostic glossary: every code UAM can report, with What it means, Where it comes from, Possible causes and What to do; searchable (spaces match underscores), the codes the tenant reports now first, codes of earlier versions kept and marked. Then ready-made searches: tracker errors, failed or incomplete runs, every run, skipped scheduled reports, REST calls, who changed UAM.

The view and its filters, and the expanded row, survive a tab switch and the auto-refresh. A diagnostic code is a chip: its short meaning on hover, its full explanation on click; it keeps its warning or failure colour only while its condition is current, and a code of a recovered past run is drawn neutral and names the run it describes. The Status verdict answers in seconds; the 24-hour history and the record counts follow in a second request.

Configuration Guardian

The uam_collection_health check (tenant scope, warning, see Configuration Guardian) runs at every health cycle of each UAM tenant and on Guardian Run now. It is skipped for a disabled tenant, an unconfigured workspace or manual collection (which clears a stale alert). The alert names the tenant, lists its problems in the metadata and points at Collection health; it clears itself once the conditions are gone. Unreadable inputs leave an existing alert untouched.

Problem code	Condition
report_missing / report_not_scheduled	A report the tenant expects is absent, or not scheduled.
no_run_recorded	A report has no recorded run.
tracker_stale	The last run is older than three missed schedule opportunities (plus the listing budget for the catalog reports).
repeated_failed_runs	Three or more consecutive failed runs of one report.
notification_delivery_failed	Operator notifications are on and the last policies run failed to deliver some.
user_notification_delivery_failed	User notifications are on and the last policies run failed to deliver some.
listing_overdue	A catalog was last published more than twice its cadence plus a day ago.
listing_incomplete	The last listing or catalog traversal failed or did not complete.
listing_queued_repeatedly	A catalog of a tier waited for another listing of the same tier on three due runs in a row and listed nothing.
checkpoint_stalled	A checkpoint did not move for three schedules plus one window — the audit activity cursor, the resource usage cursor or an account signals scan. A checkpoint catching up, however far behind, is not stalled. The alert names the cause when one is recorded: a range being written over several runs, or a held range.
activity_window_held	An audit range has been held (its search keeps failing) for 15 minutes.
window_reread_overdue	A window read while some search peers did not answer has waited more than a day to be read again.
window_recount_overdue	A recount of closed ranges has waited more than a day past its time.
findings_working_set_large	One policies run can change more than 50,000 findings. Every finding is still evaluated.
dead_job_lock	A lock is held by a run whose search has stopped (use Recover stopped run).
lock_owner_unverifiable	A lock’s owner cannot be verified (another member, or no search id).
definition_capacity	More custom policies are enabled than the tenant allows.
schedule_drift	A custom policy’s scheduled report is missing, unscheduled or differs from its published revision.
repeated_incomplete_runs	Three or more consecutive incomplete runs of a custom policy.
recovery_required	A custom policy run outlived its time limit and needs a verified recovery.
missed_schedule	A custom policy missed its scheduled runs.

Tenant lifecycle

Disabling the tenant un-schedules its reports; enabling it reconciles them. Deleting a UAM tenant cancels a running backfill, disables and deletes its reports, lookups and collections, and refuses to run while a job lock is held (a provably stopped lock is recovered first; wait for or recover a live one, then retry). When provisioning is incomplete, the wizard result and the workspace offer Repair UAM workspace, which re-provisions the collections, lookups and reports idempotently.

Settings reference

The UAM configuration is one record (uam_config at creation, config of POST uam/configure afterwards), edited from the workspace Settings tab and saved at once with a revision check (an older draft is refused). Every limit is a tunable with a validated range, not a product cap, and every limit of a run is a time: there is no row limit. The seven record-count settings of earlier versions (activity_max_records, activity_max_window_retries, resources_max_records, inventory_max_records, findings_max_records, backfill_max_records, backfill_max_retries) are ignored: stored values disappear at the next save, and an API client that still sends them keeps working.

Section	Setting (key)	Default	Bounds and meaning
Environment	Search head tiers (tiers)	["local"]	1–10: local and / or remote account names, one indexing layer.
	Route account for indexing-layer searches (account)	local	One of the tiers; local when selected, else the first tier.
Activity scope	Include / exclude (search_head_scope.include / .exclude)	empty / empty	Up to 20 hostname patterns each (*, ?); exclusions win.
Collection	Scheduled trackers (scheduled)	On	Off = collection only on request. Requires a tenant owner.
	Store SPL search (store_search_text)	Off	Index the SPL of reported searches in the search text index.
	Collect search resource usage (resources_enabled)	On	Required by the resource policies.
	Source version assumption (source_version_assumption)	On	Parse unverified emitters under the head’s version, labelled assumed.
Indexes	indexes.evidence / .findings / .receipts / .search_text	empty	Empty = the tenant’s summary index (evidence, receipts, search text) or notable index (findings).
Limits · activity	Activity window (activity_window_seconds)	300	60–3,600 s: the largest span one audit or resource window reads (a busier window is split and read in full).
	Activity overlap (activity_overlap_seconds)	60	0–600 s, less than the window.
	Audit activity max seconds per run (activity_max_seconds)	270	30–900: the budget of one Audit activity run.
	Resource usage max seconds per run (resources_max_seconds)	270	30–900: the budget of one Resource usage run, capacity refresh included.
	Account signals max seconds per run (signals_max_seconds)	870	30–900: the budget of one Account signals run (half at most for the login scan).
	Data usage run budget (data_usage_max_seconds)	870	30–14,000: the budget of one Data usage run; a scheduled run ends before the next 15-minute slot, a manual run can use the whole budget.
	Activity policy window (activity_policy_window_seconds)	900	300–86,400 s: the window of collected evidence the activity and resource policies evaluate.
Limits · inventory	Inventory page size (inventory_page_size)	500	50–500 records per REST page.
	Catalog collection every (h) (inventory_reconcile_hours)	24	1–168: saved-search listing and the ten other families.
	Users and roles every (min) (inventory_identity_minutes)	60	15–1,440.
	Full listing budget (s) (inventory_listing_max_seconds)	3,570	300–14,000 per run of a traversal; a longer traversal continues at the next run.
	Listing window start / length (inventory_listing_start_hour / inventory_listing_window_hours)	1 / 5	0–23 / 1–24 (24 = any time), search head local time.
Limits · policies	Resource rules: minimum history (h) (activity_min_history_hours)	24	0–720; the learning period of the resource policies.
Limits · backfill	Backfill days (backfill_days)	30 (the wizard proposes none)	0–90 (0 = no backfill). The wizard always sends its choice; 30 is what a stored configuration without the setting reads.
	Backfill bucket (backfill_bucket_seconds)	300	300–3,600 s.
	Backfill max seconds per run (backfill_max_seconds)	3,570	600–3,570.
Account classification	Detection (classification_detect_interactive / _service)	On / On	Human and service detection.
	Rules (classification_rules)	none	Up to 100 ordered exact / wildcard rules.
	Manual classes (account_classes)	nobody, splunk-system-user, admin → system	Up to 10,000 accounts.
Data usage	Supply links (data_usage.links)	automatic	Per tier: one VOL tenant (vol_tenant) and up to 20 DSM tenants (dsm_tenants). A tier without an entry is matched automatically; an explicit entry, even empty, stays as set.
	Excluded apps (data_usage.exclude_apps)	trackme, Splunk_SA_CIM	Up to 10 apps whose searches are shown, never counted as reads.
	Internal indexes (data_usage.include_internal)	Off	Show the _-prefixed indexes in the views by default.
	Account and sourcetype grain (data_usage.per_account / .sourcetype_facts)	On / On	Write facts per account and per sourcetype. Changes apply to new facts only.
	Resolution limits (data_usage.limits)	65,536 / 128 / 256 / 64	SPL excerpt characters (1,024–262,144), reference depth (1–256), resolution attempts per search (1–4,096) and nested subsearch levels (1–128). A limit reached leaves the search unresolved, never dropped.
Privileged users	privileged.role_patterns / .roles_include / .roles_exclude / .capabilities	*admin* / none / trackme_admin / nine capabilities	Up to 500 entries each; see Privileged users.
Notifications to operators	notifications	Off; new, reopened, severity raised, resolved; medium; digest	Up to 50 recipients.
Notifications to users	user_notifications	Off; human accounts; baseline	50 per run (1–200), 3 per address per hour (1–20), 24 h between repeat detections (1–168).
Policies	policies.<policy id>	see the catalogue	Enabled, severity, exclusions, scope, thresholds, automatic thresholds, preview mode, bank holidays hold (hold_on_bank_holidays); the privileged policies’ settings (inactive_days, reference_accounts, reference_lookups, report_stale, include_splunk_operated).

Stored elsewhere:

• AI Agent — the ai_uam_agent_* fields of the tenant (see The UAM AI Agent).

• Assignment — up to 100 assignment groups and 100 routing rules; group emails off by default.

• Finding actions — up to 50 definitions and 100 automation rules.

• Custom policies — their own definitions (see Custom policies).

• Per viewer — Exclude TrackMe activity on the Users tab (browser storage, off), the Policies view mode (Cards, Compact cards, List), and the three Tenant Home display preferences (Collection monitors, Single stats, Charts).

The tracker schedules, the one-day retention of old catalog generations and the 90-day retention of the AI Agent decision log are fixed.

Roles and licence

Role	Access
TrackMe administrator of the tenant (trackmeadminoperations and one of the tenant’s admin roles; Splunk admin_all_objects also passes)	Everything: the workspace in Tenant Home, triage and grouping, configuration, policies and custom policies, search text and live definitions, the knowledge-object catalogs, collection control, the AI Agent.
Power user of the tenant (trackmepoweroperations and one of the tenant’s power roles)	REST API access only (Tenant Home shows the UAM workspace to tenant administrators): the workspace, findings, Finding groups, counts, activity, accounts, privileged users, inventory, saved search view, policy evaluation and history, tracker history, AI Agent settings and decision log; Send email communication; findings actions (run an action, email action, set an assignment).
Read-only user (trackme_user only)	No UAM operation.

Licence: any licence beyond the Foundation edition — Enterprise, Unlimited, the trial and Developer licences. Foundation and the Foundation Trial are refused (uam_not_available_in_foundation). On an expired licence the workspace stays readable and changes are refused; scheduled collection and custom policy runs continue through the read-only and escalation stages of the licence enforcement ladder and stop with the tenants at the stop stage.

REST endpoints

Resource group uam: /trackme/v2/uam/<operation> (browse it in TrackMe REST API Reference). Every operation is a POST with tenant_id in the body, except GET workspace and GET agent_settings, and answers describe=true. “Power” marks the operations also open to tenant power users; the others need a TrackMe administrator of the tenant. Every earliest / latest accepts what the Splunk time picker produces.

Area	Operations
Workspace reads (power)	workspace, finding_counts, activity, timeline, accounts, account, privileged (the privileged users, with activity: true for their activity), inventory, saved_search, tracker_history
Data usage (power)	data_usage (view: map, the default, with the value view, or timeline for an index or an account), data_usage_index (one index), data_usage_objects (the knowledge objects), search_jobs_cancel; data_usage_links (the supply candidates, administrators). The reads accept search_jobs: "start" and then the returned jobs to poll; without it they answer synchronously within the safety timeout
Findings and groups	findings, finding_groups, finding_group (power); triage (decisions on up to 500 Finding groups, expected_versions), group_members (merge, add, remove, ungroup), classify (manual classes of 1 to 200 accounts)
Policies	policy_evaluation, policy_history (power); preview (evaluate a draft configuration without writing)
Custom policies	custom_policies, custom_policy, custom_policy_health, custom_policy_write (create, save, restore, publish, enable, disable, archive, delete, delete a version), custom_policy_execution (preview or run: dispatch, status, cancel, recover)
Notifications	notification_test (operators or users channel); user_notification_preview (power); user_contact_override; communication_preview, communication_send (power)
Inventory and catalogs	saved_search_definition (Fetch definition), knowledge_objects, search_text (Fetch SPL)
Collection	configure (full configuration with its revision), collect (job: all or one job, optional tier), backfill (pause, resume, cancel, restart), recover, collection_health (optional parts: status and / or history), storage_counts, searches (the exact searches of a tracker’s last run, job = activity, resources, signals or policies, or of one policy, policy_id)
AI Agent	agent_settings (GET power, POST saves), agent_decisions (power), agent_findings, agent_evidence, agent_decide, agent_review, agent_run; job control in /trackme/v2/ai_findings_agent/{agent,agent_status,agent_cancel}
Findings actions	/trackme/v2/findings_actions/* with component=uam

The tenant is created with POST /trackme/v2/vtenants/admin/add_tenant carrying tenant_uam_enabled=true and a uam_config object (the wizard sends tiers, scheduled, backfill_days and search_head_scope); see Creating a splk-uam tenant.

Upgrading to 2.4.22

• Data usage arrives with its own report, trackme_uam_data_usage_tracker_tenant_<tid>, created on an upgraded tenant by the health tracker’s schedule reconciliation, without a configuration save. Its history starts at its first run: the searches before the upgrade are not read. Until the collection covers the selected period, or seven days of it, the views say so far and call no index unread.

• The facts’ index-time fields (uam_data_*) are extracted by the same TrackMe ingest rule as the other UAM dimensions: on a distributed deployment, the instances that parse the UAM log need the upgraded app and its restart. Until then the map falls back to a slower raw read.

• Searches the UI starts on a UAM tenant run as the caller’s own search jobs, polled until done: a long range or a heavy account is read completely instead of being cut after 45 to 60 seconds.

Upgrading to 2.4.21

• Indexing-layer trackers: the health tracker’s report check (every 5 minutes) creates the Resource usage and Account signals reports of an existing tenant, with its owner, ACL and schedule, before the Guardian check of the same cycle (a report it just created is never reported missing). The Activity report keeps its definition and runs the audit only. Resource usage continues from the existing resource cursor, the scans from where the earlier runs stopped; until the first Resource usage run, the policies read the resource usage of the last activity receipt. No schema migration, no settings save.

• Budgets: a stored budget still at its earlier default (200 s for the activity, 3,600 s for the full listing, 3,000 s for the backfill) moves to the new default (270 s, 3,570 s, 3,570 s); a value an administrator chose is kept. The report check also refreshes a tracker description left by an earlier build.

• Retired limits: the seven record-count settings are ignored and dropped at the next save.

• Evidence: correlated executions are no longer indexed, and the UAM events carry fewer fields (see The evidence). Rewrite your own searches on trackme:uam:execution. The index-time uam_* fields of audit and resource records now follow their sourcetype: the rule takes effect at the splunkd restart, and the earlier rule still applies until then.

• Backfill: the wizard proposes no backfill and new configurations use 5-minute buckets; an existing tenant keeps its bucket size until you change it.

• Catalogs: older catalog generations are deleted after 30 minutes (a day before), which removes many KV records on a deployment with many users. The users catalog also collects last_successful_login, kept outside the compared fields (a login is never an inventory change).

• Privileged users: the tab and Settings → Privileged users appear with the release defaults; the four privileged policies are off, and a change policy turned on starts from a baseline.

Upgrading from 2.4.18

• Finding groups: the first policies runs after the upgrade consolidate the existing findings into Finding groups (up to 5,000 findings per run; an account is never split). Triage decisions, notes and history are kept.

• Inventory: each tier gets its listing and kos_* reports. The first saved-search listing runs at the next hourly wake, then on the catalog cadence. The users and roles reports move from 5-minute to 15-minute wakes unless an administrator changed their schedule. Inventory max seconds, the fallback listing cadence and the delta object limit of 2.4.18 have no effect any more.

• UAM AI Agent: the UAM Findings Advisor report is removed; an enabled advisor’s settings (enabled, mode, provider, review interval, runtime capped at 3,000 seconds) are carried to the UAM AI Agent once.

• The indexed uam_* dimensions take effect after the splunkd restart of the upgrade.

Troubleshooting and known limits

Start with Collection health (each tracker’s last run, diagnostics and recovery) and the Configuration Guardian check uam_collection_health. The technical logs carry outcomes only, with tenant_id index-time extracted: index=_internal sourcetype=trackme:custom_commands:trackmeuam* tenant_id=<tid> (trackme_uam_tracker.log, trackme_uam_inventory.log, trackme_uam_activity.log) and index=_internal sourcetype=trackme:rest_api tenant_id=<tid> (trackme_rest_api_uam.log). trackme_uam_events.log is the evidence, not a troubleshooting log.

Symptom	What to check
The inventory is complete but the Activity tab stays empty	The route account cannot read _audit on the indexing layer (roles on the remote deployment), the search heads keep their internal indexes local, or the activity scope excludes them. Check the activity receipt in Collection health and run index=_audit action=search as the route account.
Data usage calls nothing unread, and says so far with a date	The collection is younger than the selected period (or than seven days): an absence of reads is claimed from that date. The coverage card of the view gives it.
Data usage shows Partial coverage or no volume	The coverage card names the cause: a collection run that did not complete (see the Data usage tracker in Collection health), an index catalog that could not be read, facts without the rollup key (the parsing tier lacks the upgraded app), or no VOL / DSM tenant linked to the tier (Settings → Data usage).
Most audit lines are unparsed (unsupported_source_contract)	The emitters’ version could not be verified and Source version assumption is off: turn it on.
No service account is ever detected	The _audit trail is JSON-only (the automation scan reads the legacy key=value format), or its retention is under seven days. Classify the accounts by rule or by hand.
The resource policies say learning for days	Less than 24 hours of resource history (Collect search resource usage off, or _introspection not forwarded), or no capacity record (see the activity receipt).
A policy opened a finding during a bank holiday	The policy keeps opening findings (During bank holidays), the tenant does not hold new findings (Manage Bank Holidays → Tenants & settings), the holiday is not in the tenant’s calendars, or the finding was already observed. The policy insight and the policies receipt say what the run held, or that the holidays could not be read.
Peak memory reads not measured	No introspection sample caught the search (short pooled searches on Splunk Cloud). CPU still comes from Splunk’s completion accounting.
Fetch SPL shows a reason instead of the text	not_in_audit: the record aged out of _audit; account_access_denied: your roles are not granted the route’s remote account; access_denied: the environment refused your search. Store SPL search keeps the text in an index you control.
A finding keeps coming back after Resolve	Expected: the condition persists and the policy reopens a resolved finding. Fix the cause, add an owner / account or app exclusion on the policy, or dismiss it.
A CSV finding names the recorded owner, not the writer	No supported audit evidence of the write since monitoring began (or the tier’s members could not be discovered: see the policy evaluation’s writer diagnostics). The owner fallback is by design.
The Users tab shows a class the settings do not	The class is detected; a manual class or a rule always wins, and a detection can be switched off.
Collect now answers that a run is in progress	A scheduled run holds the lock; wait for its receipt. If no run is active, Recover stopped run (after five minutes, same search head only).
The backfill reports waiting for the first activity run	Normal until the Audit activity and Resource usage trackers set the live start; the backfill plans itself at its next hourly slot.
A custom policy does not run	Check Collection health → Trackers, family Custom policies: published and enabled, Scheduled trackers on, a tenant owner set; the Guardian names schedule_drift, missed_schedule or recovery_required.
The activity checkpoint lag keeps growing	The audit trail grows faster than one run reads it within its budget. Restrict the Activity scope to the search heads you monitor (it is applied in the base searches); Searches of the last run on the Audit activity tracker shows each window and its outcome. Nothing is dropped meanwhile: the next runs continue.
Audit activity says Range held	The search of one range keeps failing: it is retried, then halved to isolate the failing span, never skipped. Open the search job and the tracker log from the Status row; a range held for 15 minutes raises activity_window_held.
Windows to read again (informational)	Some search peers did not answer while a window was read. The window is read again once every peer answers, writing only what was missing; after a day of waiting, window_reread_overdue is raised: check the peers’ state.
The Duration of a run says its budget was capped	The scheduler dispatched the run late (a busy scheduler): the run ended 30 seconds before the next scheduled run so that none is skipped, and the next runs continue through the checkpoint. Frequent caps point at scheduler load on the search head.
Lookup sizes refused (lookup_size_access_denied)	On Splunk 9.x the tier’s collection account is not allowed to read admin/file-explorer, where UAM reads the CSV lookup sizes; the CSV lookup footprint policy cannot judge that tier until it is.
My searches on trackme:uam:execution return nothing recent	Correlated executions are no longer indexed since 2.4.21: rewrite them on trackme:uam:observation or read them through the activity endpoint (kind=execution).
The tenant cannot be deleted (uam_job_running_or_recovery_required)	A tracker or backfill run is live or its lock is held by another member; wait for it or recover the lock on its member, then retry.

Known limits.

• One indexing layer per tenant: keeping the tiers on one layer is the administrator’s responsibility.

• A saved search with the same name, app and owner on two tiers shares one set of run figures in the saved search view (resource records carry no tier).

• The knowledge-object catalogs reflect what the collection account can see, and a traversal is not an atomic snapshot.

• Several tenants monitoring the same tier each pay their own collection cost.

• Activity policies judge 15-minute windows of collected evidence by default; inventory policies follow the catalog cadence (a daily saved-search listing by default).

• Custom policies map to accounts and saved searches only, and their findings are not notified.

• A line that becomes searchable more than about ten minutes after its range closed, with no Splunk message, is not recovered; the login and automation scans, the resource usage and the backfill are not recounted. Rows recovered by a re-read after their policy window was evaluated are not evaluated again by the activity policies.

• A one-second range holding 500,000 audit events or more is held.

• A single Splunk call that blocks on a saturated instance is not interrupted inside a run, so such a run can last longer than its budget.

See also

• User Activity Monitoring — the task-oriented guide.

• Findings, Finding groups and findings actions — findings, Finding groups, assignment, notifications, findings actions and the AI Agents.

• Who is doing what on your Splunk? User Activity Monitoring (UAM) — the end-to-end walkthrough.

• Creating a splk-uam tenant — the creation wizard and the REST flags.

• Configuration Guardian — the Configuration Guardian.

• Remote Splunk deployments — remote deployment accounts (the remote tiers).

• The AI Advisors — AI Advisors and Agents.