Splunk UF Fleet reference

The reference of the Splunk UF Fleet tenant: every built-in policy with its defaults and settings, custom policies, the collection jobs, the settings, Collection health, the AI Agent, the indexed finding events, the permissions and the commands. The guide is Splunk UF Fleet.

Built-in policies

Twenty-eight policies in four categories. Every policy is evaluated on each live window (five minutes). Common settings of every policy:

• Enabled and Severity (Informational, Low, Medium, High, Critical; a few policies raise it, as their row says).

• Checks to open and Checks to recover: consecutive failing, then healthy, live windows.

• Scope (Groups): empty means every agent, the fleet and every group; chosen groups judge those groups and the agents they hold, never the fleet (an expected agent never seen is judged whatever the scope).

• During bank holidays: keep opening findings, or hold (Agent disconnected, Mass disconnect, Fleet coverage and Phonehome drop hold by default; see Bank holidays). Expected agent missing keeps opening: those agents were selected as always connected.

General rules:

• An incomplete read may open a finding, never close one; absence of evidence never recovers.

• A finding not evaluated for 24 hours closes as not observed.

• At most 250 new agent findings open per run (the rest the next runs).

• The troubles read from the agents’ own logs (output, destinations, throttle, latency, input errors, restarts, handshake) are decided for a connected agent only, on at most 500 agents per run (the worst first).

Agents

Policy	What it detects	Default	Severity
Agent disconnected	No receiver saw a connection from the agent in the connected window. Held while the fleet or one of its groups has a Mass disconnect; becoming dormant closes it as dormant.	On	Medium
Expected agent missing	An agent of the expected selection (typed names, lookup tables) is not connected, including a name never seen. Dormancy never closes it.	On	High
Agent output blocked	The agent paused its output (TcpOutputProc) for at least the blocked seconds in the last 15 minutes, or its health report says its output rate is red. Held while 30 % or more of the connected agents (and at least 3) are blocked together.	On	High
Agent destinations failing	The agent fails to reach one of its receivers without a break for the open delay while the other agents reach it (a receiver many agents fail on is that receiver’s Destination unreachable finding).	On	Medium
Agent throttled	A ThruputProcessor “has reached maxKBps” line in 15 minutes, three live runs in a row, and the agent’s limit is at most the reported limit (an agent rate-limited higher on purpose opens nothing).	On	Medium
Agent ingestion latency	The agent’s health report says Ingestion Latency is red in its last report period.	On	Medium
Agent input errors	TailReader warnings and errors naming at least the minimum of distinct files in 15 minutes (permissions, checksum, read errors).	On	Low
Agent restart loop	At least the minimum of starts within an hour, on the larger of the agent’s own starts (“Splunkd starting” lines, less the restarts its Deployment Server asked for) and the restarts its Deployment Server asked for: a deployment wave restarts an agent once, so fewer waves than the minimum never open it. A loop its Deployment Server causes is held while that server’s Restart storm is measured failing or not yet recovered (one finding, never one per agent); a crash loop is never held.	On	Medium
Agent identity conflict	One GUID connected from several host names at the same time (latest connection lines within 3 minutes), or one host name under several GUIDs at the same time from different source addresses, or one GUID from several addresses on one receiver within 30 seconds, again and again (a clone that kept the golden host’s name too). A rename, a replaced machine, NAT or several instances on one machine open nothing. Held while Cloned images is open.	On	High
Agent newer than receivers	The agent’s release (major.minor) is newer than the newest release of every receiver it connected to over the day. A receiver of unknown version holds.	On	Low
Agent version unsupported	The agent’s version is below the minimum, past the grace on its current version. The lowest listed version is the fleet’s minimum; a listed version of a release raises it for that release (with 9.4 and 10.0.2, 9.3.9 and 10.0.1 are below; 9.4.0 and 10.2.0 are not). Decides nothing until a minimum is set.	Off	Medium
Agent not phoning home	A Deployment Server client that forwards data has a last phonehome older than the delay, or than Splunk’s own offline rule (1.05 times its interval plus 20 s), whichever is longer. Held while its Deployment Server is silent or the agent is disconnected.	On	Medium
Deployment handshake failing	At least the minimum of DC:DeploymentClient handshake or phonehome retry lines in 15 minutes. Recovers after an hour without.	On	Medium
App deployment failed	The latest result of one of the agent’s deployments (server class, app) over the last day is a failure; a later success recovers.	On	High
Agent unmanaged	A connected agent known to no Deployment Server, while other agents have one.	Off	Low
Decommission candidate	The agent became a removal candidate (no connection for the removal delay). Reconnecting recovers; Forget removes it.	On	Low

Fleet

The fleet policies judge the fleet and every group with a known membership separately.

Policy	What it detects	Default	Severity
Mass disconnect	At least the open share of the active agents, and at least the minimum, disconnected together (fewer than 2 is never a mass disconnect). Recovers below the recover share; holds its agents’ disconnection findings.	On	High → Critical at the critical share or none connected
Fleet coverage	Connected agents below the minimum, or below the second-highest daily peak of the last 7 complete days less the drop (3 days learned at least).	On	High
Version drift	Agents of the fleet or a group below the version baseline (Settings → Versions: the most common version, the highest, or a pinned one) after the rollout grace. One cohort finding listing the versions, never one per agent.	On	Low
Cloned images	At least the minimum of agents share GUIDs at the same time (a cloned image deployed without clearing its identity): one fleet finding, holding the clones’ own identity findings.	On	High

Deployment Servers

Policy	What it detects	Default	Severity
Deployment Server silent	A Deployment Server with clients received no phonehome for 3 times its usual gap (15 minutes to 2 hours) and its deploy-server metric counted none.	On	High
Phonehome drop	The last 15 minutes below the drop percentage of a quarter of its usual hour (the median of the same UTC hour over 7 days; 3 days and 60 phonehomes an hour at least). Recovers at 80 %; held while silent.	On	High
Deployment failures	A deployment failing now, a failure in the last 15 minutes, or its failure counters rising; failures in the day but none now hold it.	On	High
Restart storm	At least the share of its clients, and the minimum of agents, restarted by it twice or more within an hour (a restart loop: an app redeployed again and again). One deployment wave opens nothing. While it is failing or recovering, its agents’ Agent restart loop findings caused by it are held.	On	Medium
Deployment Server errors	An error line of the Deployment Server’s own deployment components (server classes, deployment, apps served, client sessions) in its latest configuration load — a start or a reload, the only time it writes them. Recovers on the next load without them.	On	Medium
Requests rejected	At least the share of its phonehome and connect requests answered 4xx or 5xx in the last 15 minutes. Held under 20 requests or on a failed read.	On	Medium
Response slow	The exact 95th percentile of its phonehomes’ response time at least the threshold over the same lines. Held under 20 phonehomes or on a failed read.	On	Low

Destinations

Policy	What it detects	Default	Severity
Destination unreachable	At least the shared minimum of agents, and the shared share of the receiver’s senders, fail to reach it without a break for the open delay (longer while its cluster peer restarts). One finding on the receiver listing the agents and their count by group; the agents’ own destination findings are held.	On	High

Default settings

Each policy’s checks to open and to recover (one check per live window, every five minutes) and its own settings with their defaults.

Policy (id)	Open / recover	Settings and notes (defaults)
Agent disconnected (uf_agent_disconnected)	1 / 1	The tenant’s connected window and dormancy
Expected agent missing (uf_expected_agent_missing)	1 / 1	Host names or GUIDs (* allowed); up to 20 lookup tables; grace 0 s
Agent output blocked (uf_agent_output_blocked)	3 / 2	Blocked 60 s; shared from 30 % and 3 agents
Agent destinations failing (uf_agent_destinations_failing)	1 / 2	Open after 900 s
Agent throttled (uf_agent_throttled)	3 / 2	Lines 1; report limits up to 256 KB/s (the forwarder’s default)
Agent ingestion latency (uf_agent_ingestion_latency)	2 / 2	No setting.
Agent input errors (uf_agent_input_errors)	1 / 2	Files 3; recover after 24 h
Agent restart loop (uf_agent_restart_loop)	1 / 2	Starts 3; recover after 2 h
Agent identity conflict (uf_agent_identity_conflict)	3 / 2	Recover after 24 h
Agent newer than receivers (uf_agent_newer_than_receiver)	1 / 1	No setting.
Agent version unsupported (uf_agent_version_unsupported)	2 / 2	Minimum versions (up to 20, one per line); grace 24 h
Agent not phoning home (uf_agent_not_phoning_home)	2 / 2	Missing after 900 s (120 s to 7 days)
Deployment handshake failing (uf_agent_ds_handshake_failing)	2 / 2	Retries 10
App deployment failed (uf_agent_deployment_failed)	1 / 1	No setting.
Agent unmanaged (uf_agent_unmanaged)	1 / 1	No setting.
Decommission candidate (uf_agent_decommission_candidate)	1 / 1	The tenant’s removal delay
Mass disconnect (uf_fleet_mass_disconnect)	1 / 2	Open 20 %; at least 2; critical 50 %; recover 10 %
Fleet coverage (uf_fleet_coverage)	2 / 2	At least 0; drop 20 %
Version drift (uf_version_drift)	1 / 1	Grace 24 h
Cloned images (uf_fleet_cloned_images)	3 / 2	Agents 5; recover after 24 h
Deployment Server silent (uf_ds_silent)	1 / 2	No setting.
Phonehome drop (uf_ds_phonehome_drop)	2 / 2	Drop 50 %
Deployment failures (uf_ds_deployment_failures)	1 / 2	Recover after 24 h
Restart storm (uf_ds_restart_storm)	1 / 3	Clients 20 %; at least 5
Deployment Server errors (uf_ds_config_errors)	1 / 1	Error lines 1; count warnings off; ignored messages (globs, up to 50)
Requests rejected (uf_ds_requests_rejected)	2 / 2	Rejected 5 %
Response slow (uf_ds_response_slow)	2 / 2	p95 1,000 ms
Destination unreachable (uf_destination_unreachable)	1 / 2	Open after 900 s (1,800 s while the peer restarts); shared from 30 % and 3 agents

Custom policies

TrackMe administrators of the tenant write their own policies in the Policies tab, above the catalogue: an SPL search run on this search head as the tenant owner, on its own schedule. Each result row names one subject, which opens a finding like a built-in policy and joins the Finding group of its object.

Subject type	Result field	Resolves to
Agent	agent (a GUID or a host name)	The agent of the directory: a GUID names its agent, a host name the agents reporting it. Several agents on one name (a cloned image) are ambiguous; an agent not listed or forgotten is not monitored.
Deployment Server	ds (a host name)	Named directly.
Group	group (a group id or name, or fleet)	The group of the tenant’s settings.
Fleet	None	The fleet.

A row whose subject is not monitored is skipped and the run’s other findings recover; an unresolved row (ambiguous, or before the first publication) opens what it resolved and recovers nothing. A remote source is searched through | splunkremotesearch in the SPL. The examples offered include Agents dropping events and Deployment Servers logging errors. Drafts, published revisions and activation are managed independently, Splunk validates the SPL before every preview and run, and each policy’s runs are in Collection health. See Findings, Finding groups and findings actions.

Collection jobs

Job	Schedule	What it does
Inventory	Every 10 minutes	The agent directory from the receivers’ lines (a day), the phonehomes (a day) and the client records (30 days); the Deployment Servers and their clients’ phonehome state; each Deployment Server’s hourly phonehome baseline (the last 7 complete days); the server class and output group members; the enrichment; the telemetry probe. Writes an agent row only when something about it changed; the directory is rebuilt about hourly, always on a manual run.
Live	Every 5 minutes	The connected set, heartbeats, the agents’ own troubles, the Deployment Server reads, the groups’ membership and counts; publishes the fleet summary, writes the series, then evaluates every policy.
Policies	On demand	Evaluates the policies on the latest live publication.

Schedules are offset per tenant. Collect now (the status strip) runs the live job; Run now in Collection health runs a job; Scheduled collection (Settings → Collection) is on by default. The reports are trackme_splunk_uf_<job>_<generation>_tenant_<tenant>, and the AI Agent report trackme_uf_agent_tracker_tenant_<tenant> (scheduled only once the scheduled triage is enabled).

Settings reference

Settings is one browser with a single Save settings. A tenant role that is not a TrackMe administrator of the tenant sees it read-only. Changing a field marked starts a new source restarts collection, and the findings of the previous source close. Every policy threshold is a policy setting; the expected agents are set in their policy.

Section	Setting	Default	Notes
Source	Source (search tier)	local	Fixed once the tenant exists: create another tenant for another deployment.
	Internal, Metrics, Introspection index	_internal, _metrics, _introspection	Fixed once the tenant exists; an empty Metrics index means _internal only.
	Telemetry form	Automatic	Automatic, _metrics mirror, or _internal; starts a new source.
	Agent patterns	*	Host name patterns (* and ?), matched case-insensitively. Every pattern setting of this section takes one token per host name or pattern (type it and press Enter; a pasted comma-separated list is split).
	Receiver patterns	*	The receivers whose connection lines count: host names or patterns with * (the only wildcard here). Every read of the receivers’ lines takes it.
	Excluded agents, Excluded receivers	None	Never * alone; Excluded receivers take * as their only wildcard. Set at creation, the first backfill never reads what they leave out. A receiver left out stays infrastructure, never an agent.
	Full instances	Left out	Counted as agents makes every full-instance leaf sender an agent.
	Repair workspace	—	Recreates the tenant’s collections and reports; findings and settings are kept.
Deployment Servers	Deployment Server source	local	Where the _ds* indexes are searched: a remote account on the Deployment Server on Splunk Cloud.
	Deployment Server API accounts	None	Optional, one per Deployment Server (up to 20), for the exact server class membership; never a new source. Its role needs list_deployment_server, and must be allowed to search _dsphonehome, _dsappevent and _dsclient: Splunk builds the client listing from these indexes, searched as the account (else the check reads Cannot search the DS indexes).
	Phonehome, Client, App event index	_dsphonehome, _dsclient, _dsappevent	Starts a new source.
Connectivity	Connected window	15 minutes	5 to 60.
	Heartbeat fallback	On	The heartbeat never creates an agent.
	Dormant after, Removal candidate after	7 days, 30 days	1 to 90; 1 to 365 and at least the dormancy delay, counted from the last connection.
Enrichment	Lookup tables and their fields	None	Up to 5 lookups of up to 500,000 rows, 10 fields each; never a new source.
Groups	Groups	None	Up to 200, 20 of them by lookup table.
Versions	Version baseline	The most common version	The most common, the highest, or a pinned version (Version drift).
Collection	Scheduled collection	On
Notifications, AI Agent, Assignment, Finding actions	Shared settings of the findings tenants	Off / none	See Findings, Finding groups and findings actions; the assignment criterion is the agent group.

Collection health codes

The Collection health tab and the Configuration Guardian check cribl_collection_health (warning) report:

Code	Severity	Meaning
live_never_ran, live_not_running, live_failing	Warning, Failure, Failure	No live run since configuration; none for 20 minutes; runs not completing for 30 minutes.
inventory_never_ran, inventory_not_running, inventory_failing	Warning, Failure, Failure	No inventory run; none for 3 hours; not completing for 90 minutes.
policies_failing	Failure	Policy evaluation not completing for 30 minutes.
no_forwarding_receivers, no_agents	Warning	An hour after configuration, the source shows no receiver of forwarded data; or the directory holds no agent.
trouble_read_failed, trouble_read_partial, trouble_hosts_truncated	Warning	The agents’ own-log reads failed, came back partial, or more than 500 agents had something to report.
ds_read_failed, ds_read_partial, ds_clients_truncated	Warning	The Deployment Server indexes unreadable, partial, or more clients to report than kept.
ds_baseline_failed, ds_recent_failed, ds_resources_failed, ds_requests_failed, ds_errors_failed	Warning	The hourly baseline, the last 15 minutes’ phonehomes, the CPU and memory, the access log, or the configuration load errors could not be read.
identity_addresses_failed, receiver_versions_failed	Warning	The agents’ source addresses (identity conflicts) or the receivers’ versions (Agent newer than receivers) could not be read.
group_lookups_failing, group_reads_failing	Warning	A lookup group’s lookup gave no value; the server class or output group membership read failed or was cut.
group_serverclass_listing_unused	Warning	Deployment Server API accounts are configured but a client listing could not be used, or no account answers for a Deployment Server of the fleet: server class groups count app events only.
expected_lookups_failing	Warning	An expected-agents lookup gave no host name (the last good values are in use).
notification_delivery_failing	Warning	Notification emails failing.
custom_*	Warning	Custom policies: schedule drift, failing runs, missed schedules, recovery required.

AI Agent

The Splunk UF Fleet AI Agent settings (Settings → AI Agent, or Manage AI Agents):

Setting	Default	Notes
Scheduled triage	Off	Hourly when on (ai_uf_agent_enabled).
Mode	Inspect	Inspect writes assessments and notes; act also applies the decisions allowed below.
AI provider	First available
Findings per run, time budget per run	10, 3,000 s
Review interval	1 day	A finding reviewed by a scheduled run is skipped that long unless it moved.
Allow investigating, allow temporary dismissal, allow resolve	Yes, yes (up to 7 days), no	A scheduled run never overrides an operator decision; it never Forgets an agent nor adds one to the expected agents.
Custom instructions	None	Added to the agent’s instructions; Evaluate… tries them on a finding.

Runs are audited with the sourcetypes trackme:ai_agent:uf_agent:inspect and trackme:ai_agent:uf_agent:act. See AI Agents for findings.

Indexed finding events

Every finding transition is written to the tenant’s notable index as sourcetype= trackme:cribl:finding (and Finding groups as trackme:cribl:finding_group), with object_category=splunk_uf, whatever the notification settings. The tenant’s fleet, group and Deployment Server series are in its metric index under trackme.uf.*.

Permissions

• Any role of the tenant reads the workspace.

• The tenant’s power and admin roles triage findings and Forget agents.

• A TrackMe administrator of the tenant (trackmeadminoperations and an admin role of the tenant) edits Settings, policies, groups, custom policies and the expected agents (Add to expected is a policy save).

• The scheduled runs search as the tenant owner: every remote account and lookup table the tenant reads must be usable by the owner.

Commands

Command	What it does
| trackmeufdiscover	The wizard’s discovery (read-only, as the caller); mode=ds_api ds_api_accounts=<a,b> checks the Deployment Server API accounts.
| trackmeuftrend	24-hour series (kind=fleet|group|ds|agent): TrackMe’s series for the fleet, a group or a Deployment Server; Splunk’s own lines for one agent (at most 14 days).
| trackmeufrecent	The Last 5 minutes figures (never a state, writes nothing).
| trackmeufevidence	The raw lines behind a figure of an agent or a Deployment Server (range=3600|14400|86400): a timeline, a breakdown and at most 50 lines.