Splunk S2S reference

The reference of the Splunk S2S tenant: every built-in policy with its defaults and settings, custom policies, the collection jobs, the settings, Collection health, the AI Agent, the indexed finding events, the permissions and the commands. The guide is Splunk S2S.

Built-in policies

Twenty-two policies in four categories. Every policy is evaluated on each live window (five minutes). Common settings of every policy:

• Enabled and Severity (Informational, Low, Medium, High, Critical; a few policies raise it, as their row says).

• Checks to open and Checks to recover: consecutive failing, then healthy, live windows.

• Scope (Pools): empty means every pool of the tenant and the hosts no pool claims.

• During bank holidays: keep opening findings, or hold (Pool traffic drop and Pool traffic outlier hold by default; see Bank holidays).

General rules:

• A partial window (a capped read, search peers that did not answer, a failed second read) may open a finding, never close one.

• The pipeline, output and resources policies decide only when those reads succeeded.

• A finding not evaluated for 24 hours closes as not observed.

• Hold during planned work (Settings → Pools, on by default) turns a failing decision of Receiver silent, Pool capacity, Pool not receiving, Pool traffic drop, Receiver imbalance, Connection churn and Pool traffic outlier into a hold on a cluster pool in maintenance mode or a rolling restart.

Receivers

Policy	What it detects	Default	Severity
Receiver not accepting	The receiver logs TcpInputProc “Stopping all listening ports” (its queues stayed blocked) with no “Started listening” since. Recovers on the restart line.	On	Critical
Receiver silent	No connection line in the live window while other members of its pool have lines (a whole pool silent is Pool not receiving, its members held). A replaceable pool waits for a replacement first.	On	High
Receiver back pressure	A queue of the receiver path saturated (90th-percentile fill ≥ the saturation, or blocked in ≥ the blocked share of its samples). The finding names the bottleneck and whether its thread was busy.	On	High
Receiver imbalance	In a pool of at least 3 active members, a member’s share of the pool’s senders (or volume) is below the fairness threshold times an even split.	On	Low
Connection churn	Reconnections per sender per hour at least 3 times the median of the pool’s other members, and at least the minimum in the window. Recovers below 1.5 times.	On	Medium
Acknowledgment posture	A cluster peer receives senders with ack=false where acknowledgement is required (Settings → Posture: Required, or Automatic and most senders of the pool use it).	On	Low
TLS posture	Senders connect to a receiver with ssl=false where TLS is required (Required, or Automatic and the pool’s majority is TLS).	On	Low
Version best practice	A sender of the day runs a newer major.minor version than the receiver (Splunk’s matrix allows it; its best practice is receivers at least as new as every forwarder). Unknown while the receiver’s version is unknown.	On	Low

Intermediates

Policy	What it detects	Default	Severity
Intermediate output blocked	The intermediate’s TcpOutputProc paused the data flow for at least the blocked seconds in the window. Counts the senders held and names the destinations; an output queue still at 80 % or more keeps it open. Held while 30 % or more of the intermediates (and at least 2) are paused together: the next tier is the problem, and the Back pressure cascade and Destination unreachable findings say which.	On	Critical
Intermediate back pressure	A queue of the intermediate path saturated. Names the origin: downstream (its output paused, or a receiver pool it feeds backs up), upstream surge (inbound above 1.5 times its learned volume), else local.	On	High
Intermediate silent	As Receiver silent, within the intermediate pool.	On	High
Intermediate destinations failing	The intermediate fails to reach one of its destinations without a break for 15 minutes while the other senders reach it (a destination many senders fail on is that destination’s finding).	On	Medium
Intermediate single destination	An output group names one destination for the minutes set while it named two or more in the last 24 hours.	On	Medium
Intermediate in / out anomaly	What it receives and what it forwards drift beyond the factor (or below its inverse) of their learned ratio (the median of up to 7 daily medians, 3 days learned), without an output pause.	On	Medium
Intermediate TLS posture	Senders connect to the intermediate without TLS where its pool expects it (the forwarders sending to a heavy forwarder tier).	On	Low
Universal forwarder relay	A universal forwarder acting as intermediate receives universal forwarders (Splunk advises against it: partial events, blocked output queues).	On	Low

Pools

Policy	What it detects	Default	Severity
Pool not receiving	Every member of a receiver pool is silent.	On	Critical
Pool capacity	Fewer members up than expected (the cluster’s peers, else the configured size, else the size learned over 7 days). Nothing is decided while nothing is learned; a replaceable pool waits for replacements.	On	High → Critical when no member is up
Pool traffic drop	The pool’s rate is below the drop percentage of the same hour’s median over 7 days (3 days learned at least). Recovers at 80 %.	On	High → Critical when nothing is received
Pool traffic outlier	KB received (or connected senders) outside the traffic model’s range for this hour of the week, from the model confidence set, when the window or the model reaches the minimum.	On (needs a model)	Medium
Back pressure cascade	In one window, a receiver pool has a member backing up or not accepting and an intermediate is paused towards it (an intermediate tied to no pool counts towards every receiver pool). One finding names the origin and the intermediates held.	On	Critical

Destinations

Policy	What it detects	Default	Severity
Destination unreachable	At least the shared minimum of senders, and the shared share of the destination’s senders, fail to reach one receiver without a break for the open delay (longer while its cluster peer restarts). One finding on the destination listing the senders; the senders’ own destination findings are held.	On	High

Default settings

Each policy’s checks to open and to recover (one check per live window, every five minutes) and its own settings with their defaults.

Policy (id)	Open / recover	Settings and notes (defaults)
Receiver not accepting (s2s_receiver_not_accepting)	1 / 2	No setting.
Receiver silent (s2s_receiver_silent)	3 / 2	Grace 0 s; replacement grace 1,800 s
Receiver back pressure (s2s_receiver_back_pressure)	2 / 2	Saturation 80 %; blocked share 0.3
Receiver imbalance (s2s_receiver_imbalance)	3 / 2	Fairness 0.8; measure connections or volume
Connection churn (s2s_receiver_connection_churn)	2 / 12	At least 20 reconnections
Acknowledgment posture (s2s_receiver_ack_posture)	1 / 2	At least 1 sender; recover after 24 h
TLS posture (s2s_receiver_tls_posture)	1 / 2	Recover after 24 h
Version best practice (s2s_version_best_practice)	1 / 2	Recover after 24 h
Intermediate output blocked (s2s_intermediate_output_blocked)	3 / 2	Blocked 30 s; shared from 30 % and 2 intermediates
Intermediate back pressure (s2s_intermediate_back_pressure)	2 / 2	Saturation 80 %; blocked share 0.3
Intermediate silent (s2s_intermediate_silent)	3 / 2	Grace 0 s; replacement grace 1,800 s
Intermediate destinations failing (s2s_intermediate_destinations_failing)	1 / 2	Open after 900 s
Intermediate single destination (s2s_intermediate_single_upstream)	1 / 1	15 minutes
Intermediate in / out anomaly (s2s_intermediate_fan_in_anomaly)	3 / 2	Ratio factor 2.0
Intermediate TLS posture (s2s_intermediate_tls_posture)	1 / 2	Recover after 24 h
Universal forwarder relay (s2s_uif_relay)	1 / 2	Recover after 24 h
Pool not receiving (s2s_pool_not_receiving)	1 / 1	No setting.
Pool capacity (s2s_pool_capacity)	2 / 2	Replacement grace 1,800 s
Pool traffic drop (s2s_pool_traffic_drop)	2 / 2	Drop 50 %
Pool traffic outlier (s2s_pool_traffic_outlier)	2 / 2	Measure (KB received or connected senders); direction; at least 100; open from normal confidence
Back pressure cascade (s2s_backpressure_cascade)	3 / 2	No setting.
Destination unreachable (s2s_destination_unreachable)	1 / 2	Open after 900 s (1,800 s while the peer restarts); shared from 30 % and 3 senders

Custom policies

TrackMe administrators of the tenant write their own policies in the Policies tab, above the catalogue: an SPL search run on this search head as the tenant owner, on its own schedule. Each result row names one subject, which opens a finding like a built-in policy and joins the Finding group of its object.

Subject type	Result field	Resolves to
Receiver	receiver (host name)	The host in the live publication (else the directory), with its pool. A host the tenant does not follow as a receiver is not monitored.
Intermediate forwarder	intermediate (host name)	The same, as an intermediate.
Pool	pool (pool id or name)	A configured pool, a cluster or cluster site pool, an automatic intermediate pool or discovered; a name two pools carry is ambiguous.

A row whose subject is not monitored is skipped and the run’s other findings recover; an unresolved row (ambiguous, or before the first publication) keeps every finding of the run as it is. A remote source is searched through | splunkremotesearch in the SPL. The examples offered include Receivers logging S2S input errors and Intermediate forwarders failing TLS. Drafts, published revisions and activation are managed independently, Splunk validates the SPL before every preview and run, and each policy’s runs are in Collection health. See Findings, Finding groups and findings actions.

Collection jobs

Job	Schedule	What it does
Inventory	Every 10 minutes	Roles over a day, the intermediates’ type, the indexer clusters (the search head’s view every run, the manager through its account), the pools, the telemetry probe; publishes the directory (rebuilt about hourly, always on a manual run).
Live	Every 5 minutes	Index-time windows with catch-up: per host the inbound connections, senders, posture, pipeline queues, listening ports, output, health, churn, CPU and memory; per pool the sums, members up and balance. Writes the series, then evaluates every policy.
Traffic ML models	Twice an hour	Learns each receiver pool’s models when due.
Policies	On demand	Evaluates the policies on the latest live window.

Schedules are offset per tenant. Collect now (the status strip) runs the live job; Run now in Collection health runs a job; Scheduled collection (Settings → Collection) is on by default, off means manual collection only. The reports are trackme_splunk_s2s_<job>_<generation>_tenant_<tenant>, and the AI Agent report trackme_s2s_agent_tracker_tenant_<tenant> (scheduled only once the scheduled triage is enabled).

Settings reference

Settings is one browser with a single Save settings. A tenant role that is not a TrackMe administrator of the tenant sees it read-only. Changing a field marked starts a new source restarts collection, and the findings of the previous source close. Every policy threshold is a policy setting, never a tenant setting.

Section	Setting	Default	Notes
Source	Source (search tier)	local	Fixed once the tenant exists: create another tenant for another deployment.
	Internal, Metrics, Introspection index	_internal, _metrics, _introspection	Fixed once the tenant exists; an empty Metrics index means _internal only.
	Telemetry form	Automatic	Automatic, _metrics mirror, or _internal; starts a new source.
	Host patterns, Excluded hosts	*, none	Which hosts may be receivers or intermediates, one token per host name or pattern (type it and press Enter); never * alone in the exclusions; starts a new source.
	Repair workspace	—	Recreates the tenant’s collections and reports; findings and settings are kept.
Pools	Indexer clusters as pools	Automatic	One pool per indexer cluster, or per site of a multisite cluster (<cluster>:<site>, when every peer has a site and at least two sites are known). Off: clusters ignored, configured pools only.
	Intermediate forwarders as pools	Automatic	The intermediates no pool claims (no assignment, no configured pattern) form one automatic pool per host-name family. Off: they wait as discovered.
	Hold during planned work	On	The maintenance hold of the cluster pools.
	Manager accounts	None	Per indexer cluster (every site pool of the cluster uses it), checked as chosen.
	Configured pools	None	Up to 200 pools in all (automatic ones included), with up to 50 host patterns and 50 output groups each. Configure on an automatic pool makes it a configured pool with the same id.
	Auto-join, hosts assigned by hand	On, none	A newly seen host joins the pool whose patterns match; up to 500 hosts assigned by hand from each host’s pool pill, never to a replaceable pool.
Posture	TLS, Indexer acknowledgement	Automatic	Automatic (the pool’s majority), Required, Not required.
Collection	Scheduled collection	On
Traffic ML models	Model learning history, model defaults	14 days	0, 7, 14, 30, 60 or 90 days.
Notifications, AI Agent, Assignment, Finding actions	Shared settings of the findings tenants	Off / none	See Findings, Finding groups and findings actions; the assignment criterion is the pool.

Collection health codes

The Collection health tab and the Configuration Guardian check cribl_collection_health (warning) report:

Code	Severity	Meaning
live_never_ran, live_not_running, live_failing	Warning, Failure, Failure	No live run since configuration; none for 20 minutes; runs not completing for 30 minutes.
inventory_never_ran, inventory_not_running, inventory_failing	Warning, Failure, Failure	No inventory run; none for 3 hours; not completing for 90 minutes.
policies_failing	Failure	Policy evaluation not completing for 30 minutes.
models_failing	Failure	Traffic model training not completing for 90 minutes.
no_forwarding_receivers, no_s2s_hosts	Warning	An hour after configuration, the source shows no receiver of forwarded data; or the directory holds no receiver and no intermediate.
cluster_read_failed, cluster_read_refused, cluster_peers_failed	Warning	The cluster read failed (last membership kept); the reader lacks list_indexer_cluster; the peers’ read failed (maintenance mode and restart state unknown).
cluster_unreported, cluster_membership_unreadable, cluster_manager_unreadable	Warning	A cluster pool the search head no longer reports; the search head cannot reach a cluster’s manager; a pool’s manager account fails (the planned-work hold relies on the search head).
identity_read_failed, roles_unresolved	Warning	The intermediates’ type read failed; intermediates that could not be typed.
pipeline_read_failed, pipeline_read_partial	Warning	The live run’s pipeline read failed or was partial: the pipeline policies wait.
receivers_telemetry_late	Warning	The receivers’ own telemetry arrives later than the settle delay.
capacity_reached	Warning	A run reached one of its bounds (a capped read, the metric points’ bound).
notification_delivery_failing	Warning	Notification emails failing.
custom_*	Warning	Custom policies: schedule drift, failing runs, missed schedules, recovery required.

AI Agent

The Splunk S2S AI Agent settings (Settings → AI Agent, or Manage AI Agents):

Setting	Default	Notes
Scheduled triage	Off	Hourly when on (ai_s2s_agent_enabled).
Mode	Inspect	Inspect writes assessments and notes; act also applies the decisions allowed below.
AI provider	First available
Findings per run, time budget per run	10, 3,000 s
Review interval	1 day	A finding reviewed by a scheduled run is skipped that long unless it moved.
Allow investigating, allow temporary dismissal, allow resolve	Yes, yes (up to 7 days), no	A scheduled run never overrides an operator decision.
Custom instructions	None	Added to the agent’s instructions; Evaluate… tries them on a finding.

Runs are audited with the sourcetypes trackme:ai_agent:s2s_agent:inspect and trackme:ai_agent:s2s_agent:act. See AI Agents for findings.

Indexed finding events

Every finding transition is written to the tenant’s notable index as sourcetype= trackme:cribl:finding (and Finding groups as trackme:cribl:finding_group), with object_category=splunk_s2s, whatever the notification settings: use them for Splunk alerts, Splunk Enterprise Security, SOAR or dashboards. The tenant’s series are in its metric index under trackme.s2s.receiver.*, trackme.s2s.intermediate.* and trackme.s2s.pool.*.

Permissions

• Any role of the tenant reads the workspace.

• The tenant’s power and admin roles triage findings.

• A TrackMe administrator of the tenant (trackmeadminoperations and an admin role of the tenant) edits Settings, policies, custom policies, model tuning and relearning.

• The scheduled runs search as the tenant owner: every remote account the tenant reads must be usable by the owner, and the indexer cluster reads need list_indexer_cluster.

Commands

Command	What it does
| trackmes2sdiscover	The wizard’s discovery (read-only, as the caller); manager_accounts=<a,b> adds the manager account check, mode=accounts runs the check alone.
| trackmes2strend	TrackMe’s S2S series of a receiver, an intermediate, a pool or the tenant (kind=receiver|intermediate|pool|tenant), with the model band on pools.
| trackmes2srecent	The Last 5 minutes figures of the tenant, a pool or a host (never a state, writes nothing).
| trackmes2sevidence	The raw lines behind a figure for a host or a pool’s members (kind=back_pressure|posture| churn|health, range=3600|14400|86400): a timeline, a breakdown and at most 50 lines. kind=queue_history (a host only, range=900 too) reads each queue’s usage, used and capacity bytes and blocked samples: the charts of the Blocked now dialog.
| trackmes2spath	The Data path of the newest window, read with the caller’s rights (nothing saved).