Splunk Forwarder Collectors reference¶
The reference of the Splunk Forwarder Collectors tenant: every built-in policy with its defaults and settings, custom policies, the collection jobs, the settings, the input states, Collection health, the AI Agent, the indexed finding events, the permissions and the commands. The guide is Splunk Forwarder Collectors.
Built-in policies¶
Twenty-eight policies in four categories. Every policy is evaluated on each live window (five minutes). Common settings of every policy:
Enabled and Severity (Informational, Low, Medium, High, Critical; a few policies raise it, as their row says).
Checks to open and Checks to recover: consecutive failing, then healthy, live windows.
Scope (Groups): empty means every group of the tenant and the collectors no group claims (Ungrouped collectors, which can also be chosen).
Inputs not judged (the six input policies: Expected input not running, Input errors, Input went quiet, Input process stopped, Input crash loop, Input resources): up to 200 inputs this policy no longer judges, one per line, as their finding names them:
collector · inputon one collector (group · inputfor a group’s expected input), or the input alone on every collector and group;*and?are wildcards, every other character is literal. A listed input stays monitored and every other policy keeps judging it; its open finding closes as not evaluated at the next run. Set from a finding (Exclude this input from this policy…) or here. See Stop a policy judging one input.During bank holidays: keep opening findings, or hold (Collected volume drop, Collected volume outlier and Mass silence hold by default; see Bank holidays). Expected collector missing keeps opening: those collectors were selected as always reporting.
General rules:
A partial window (a read that failed, or one Splunk finalized early) may open a finding, never close one. The policies that judge an absence or a count (Collector silent, Expected collector missing, Group below minimum, Mass silence, Collected volume drop, Expected input not running, Input process stopped, Input went quiet) hold a failing decision on a partial window.
A silent collector is judged by its silence alone: every other decision on it waits.
The policies judge the latest live publication only: when it is more than 20 minutes old, they judge nothing until the live run publishes again.
A finding not evaluated for 24 hours closes as not observed.
An input is what an operator configured; a process is what runs. One script may run every stanza of its modular input, so a process names no input.
Collectors¶
Policy |
What it detects |
Default |
Severity |
|---|---|---|---|
Collector silent |
No throughput line from the collector in the live window. Stalled, not silent (held), when its output was blocked in the last 15 minutes: its own telemetry queues behind its data. Held while a Mass silence finding is open. A collector the reads no longer show stays listed as silent, with its finding, for 30 days. |
On |
High |
Expected collector missing |
A collector of the expected selection (typed names or patterns, lookup tables) is silent in the live window, or an exact name no instance of the inventory carries (never seen). Decides nothing until a selection is set. |
On |
High |
Output blocked |
The output paused ( |
On |
High |
Destinations failing |
The collector fails to reach one of its destinations without a break (failures at most 10 minutes apart) for the open delay while other senders reach it (a destination many senders fail on is that destination’s Destination unreachable finding). |
On |
Medium |
Pipeline back pressure |
A queue of the collector’s own path (parsing to output) fills to the saturation at the 90th percentile, or is reported blocked in the blocked share of its samples. Names the bottleneck queue (the furthest downstream full queue) and its origin: the output waiting on the receivers, a stage on this host, or a surge from upstream (taking in more than 1.5 times its usual rate at this hour). Held while the output block is shared or a receiver’s cluster peer restarts. |
On |
High |
Throttled |
At least the minimum of |
On |
Medium |
Ingestion latency |
The collector’s health report shows the ingestion latency indicator red (yellow is shown and opens nothing). Held while the output block is shared or a receiver’s cluster peer restarts. |
On |
Medium |
Restart loop |
At least the minimum of starts within an hour, on the larger of the collector’s own starts and the restarts its Deployment Server asked for: a deployment wave restarts it once, whatever its lines repeat. Recovers after the recovery period without a start. |
On |
Medium |
Identity conflict |
One GUID reports from several host names, or one host name from several GUIDs, with periods that overlap in time (a cloned collector image). GUIDs that follow each other on a host are a rebuild and open nothing. |
On |
High |
Version unsupported |
The collector’s version is below the minimum, past the grace on its current version. The lowest listed version is the minimum; a listed version of a release (major.minor) raises it for that release. Decides nothing until a minimum is set. |
Off |
Medium |
Host resources |
The collector’s CPU (user and system, 90th percentile over 15 minutes) or memory at or above its threshold. |
On |
Medium |
Disk space |
The fullest partition Splunk reports on the collector (its partition samples, every 10 minutes) at or below the free share: persistent queues fill it when the output blocks. |
On |
High → Critical at or below the critical free share |
Configuration changed |
A watched configuration file of the collector changed ( |
On |
Low |
Collected volume drop |
The data the collector took in over a rolling hour below the drop percentage of its usual rate at this hour (the median of the same UTC hour over the last 7 days, 3 days learned at least, learned whether or not the policy is on). Recovers at 80 %; in between, an open finding stays as it is. Held while the output block is shared or a receiver’s cluster peer restarts. |
On |
High |
Collected volume outlier |
The KB collected in the live window outside its traffic model’s range for this hour of the week (UTC weekday and hour), from the model confidence set, when the window or the model reaches the minimum. A model still learning decides nothing; a silent collector is never a drop (Collector silent judges it). |
On (needs a model) |
Medium |
Inputs¶
File input backlog, File input errors and Parsing errors are listed here and judged on the collector.
Policy |
What it detects |
Default |
Severity |
|---|---|---|---|
Expected input not running |
An input that must stay enabled (Inventory → Inputs) on a collector it names, whatever its group, or a group’s expected input on a member: disabled, or absent for the absent delay according to a source able to say so (an API listing, or the registrations since a start that was read, for a pattern naming a command). In a group whose members share their inputs: an expected input (and, with Learn the shared inputs, one a member ran in the last day) runs on no member; an input moving between members opens nothing. No source able to say reads unknown and holds, never absent. Held while the collector is silent and for 15 minutes after a start. |
On |
High; an input that must stay enabled is at least High |
Input errors |
At least the minimum of fatal and error lines over the last three live windows on the input’s stderr, counted by the level each line states itself (splunkd relays every stderr line as ERROR; a stack trace counts as one error). Or, in its add-on’s own log files, after an hour of learning: at least the minimum and twice what the input usually writes (learning stops while it fails). Held for 15 minutes after a start of the collector or of the input; recovers after the recovery period without an error. |
On |
Medium |
Input went quiet |
A signal that was regular for the input stopped: a source delivering on a steady rhythm, an add-on log file or, on syslog or file collectors, the files of a monitor stanza reaching the indexes (judged once per group). Away for three times the longest gap it ever had and at least the quiet minimum. A rhythm needs at least 12 sightings over at least a day, with long gaps (more than three times its usual one) rare, at most one per day: a late delivery, a nightly batch or a weekly maintenance keeps it watched, and the same pause never opens a finding. What was never regular is never judged, and a quiet input alone says nothing. Recovers after the recovery period. |
On |
Medium |
Input process stopped |
A long-running input process (sampled in at least the presence share of the live windows over at least 6 hours, with a process older than 15 minutes in nearly all of them) not sampled for the absent delay. A periodic input is never long-running. Held for 10 minutes after a start of the collector and while it is silent; an input excluded, or past the 50 listed per collector, is never judged. |
On |
High |
Input crash loop |
A long-running input process restarted by splunkd at least the minimum of times within the hour (its extra process ids). |
On |
Medium |
Input resources |
An input process at or above the CPU share of one core (90th percentile) or the memory size, or whose memory grew by the growth share over a reference at least 6 hours old (processes of 100 MB or more). A short process reporting no CPU reads unknown, never 0. |
On |
Medium |
File input backlog |
At least the minimum of files queued by the file monitor, rising over three live windows in a row. A steady count holds; held for 15 minutes after a start (a restart catches up). |
On |
Medium |
File input errors |
|
On |
Low |
Parsing errors |
At least the minimum of parsing warning or error lines in the last 15 minutes (truncation, timestamps, line breaking, structured formats). The finding names the top source types. |
On |
Low |
Groups and fleet¶
Policy |
What it detects |
Default |
Severity |
|---|---|---|---|
Group below minimum |
Fewer healthy members than the minimum set on the group. Healthy means reporting in the live window with its output not blocked; the health report does not count. A member blocked just before it went quiet counts as blocked. Held while members whose output was not read could make up the minimum, or while the members are blocked along with the rest of the fleet. Judges only a group that sets a minimum. |
On |
High → Critical when no member is healthy |
Mass silence |
At least the open share of the collectors, and at least the minimum, silent together in the live window. Recovers below the recover share; holds the members’ Collector silent findings. A collector the reads stopped showing earlier counts for neither side. |
On |
Critical |
Version drift |
Members of a group below the group’s version baseline (the most common version among its reporting members, the higher on a tie) after the rollout grace. One finding per group listing the versions, never one per collector. |
On |
Low |
Destinations¶
Policy |
What it detects |
Default |
Severity |
|---|---|---|---|
Destination unreachable |
At least the shared minimum of monitored collectors, and the shared share of the destination’s senders, fail to reach it without a break for the open delay (longer while its cluster peer restarts). One finding on the destination listing the senders; the collectors’ own Destinations failing findings are held. Only monitored collectors count as failing senders: a receiver only universal forwarders fail on is no finding here. |
On |
High |
Default settings¶
Each policy’s checks to open and to recover (one check per live window, every five minutes) and its own settings with their defaults.
Policy (id) |
Open / recover |
Settings and notes (defaults) |
|---|---|---|
Collector silent ( |
3 / 2 |
No setting. |
Expected collector missing ( |
1 / 1 |
Host names or patterns; up to 20 lookup tables; grace 0 s |
Output blocked ( |
2 / 2 |
Blocked 30 s; held from 30 % of the collectors and 2 collectors |
Destinations failing ( |
1 / 2 |
Open after 900 s |
Pipeline back pressure ( |
2 / 2 |
Saturation 80 %; blocked share 0.3 |
Throttled ( |
3 / 2 |
Lines 1 |
Ingestion latency ( |
2 / 2 |
No setting. |
Restart loop ( |
1 / 2 |
Starts 3; recover after 2 h |
Identity conflict ( |
3 / 2 |
Recover after 24 h |
Version unsupported ( |
2 / 2 |
Minimum versions (up to 20, one per line); grace 24 h |
Host resources ( |
3 / 3 |
CPU 85 %; memory 90 % |
Disk space ( |
2 / 2 |
Free 10 %; critical at 5 % (below the free share) |
Configuration changed ( |
1 / 1 |
Files |
Collected volume drop ( |
2 / 2 |
Drop 50 % |
Collected volume outlier ( |
2 / 2 |
Measure KB collected; direction drops and spikes; at least 100 KB per five minutes; open from normal confidence (7 days learned; low confidence: 1 day) |
Expected input not running ( |
2 / 2 |
Absent after 1,800 s; learn the shared inputs on; inputs that must stay enabled (up to 500, each on up to 50 collector names or patterns), set in Inventory → Inputs; inputs not judged |
Input errors ( |
2 / 2 |
Error lines 5; count warnings off; recover after 1 h; inputs not judged |
Input went quiet ( |
2 / 2 |
Quiet for at least 1,800 s; recover after 1 h; inputs not judged |
Input process stopped ( |
1 / 2 |
Long-running at 90 % of samples; absent after 900 s; inputs not judged |
Input crash loop ( |
1 / 2 |
Restarts 3; recover after 2 h; inputs not judged |
Input resources ( |
3 / 3 |
CPU 50 % of one core; memory 1,024 MB; growth 50 % in 6 hours; inputs not judged |
File input backlog ( |
3 / 2 |
Files 100 |
File input errors ( |
1 / 2 |
Files 3; recover after 24 h |
Parsing errors ( |
1 / 2 |
Lines 50; recover after 24 h |
Group below minimum ( |
2 / 2 |
No setting: the minimum is the group’s (Settings → Groups). |
Mass silence ( |
1 / 2 |
Open 20 %; at least 2; recover 10 % |
Version drift ( |
1 / 1 |
Grace 24 h |
Destination unreachable ( |
1 / 2 |
Open after 900 s (1,800 s while the peer restarts); shared from 30 % and 2 collectors |
Custom policies¶
TrackMe administrators of the tenant write their own policies in the Policies tab, above the catalogue: an SPL search run on this search head as the tenant owner, on its own schedule. Each result row names one subject, which opens a finding like a built-in policy and joins the Finding group of its object.
Subject type |
Result field |
Resolves to |
|---|---|---|
Collector |
|
The monitored collector of the directory, with its group. A name the directory does not list is not monitored; a GUID on several host names is ambiguous. |
Input |
|
The collector as above; the input matched against the inputs it registered, as a command line or a pattern. A name only an add-on logs (a DB Connect input name) stands for itself; a pattern naming several inputs of the collector is ambiguous. |
Group |
|
A group of the tenant’s settings. Another value is not monitored; a name two groups carry is ambiguous. |
Fleet |
None |
The fleet. |
A row whose subject is not monitored is skipped and the run’s other findings recover; an
unresolved row (ambiguous, or any row before the first directory) withholds every recovery of the
run. A remote source is searched through | splunkremotesearch in the SPL. The examples offered
are DB Connect inputs failing and Collectors whose inputs log errors. Drafts, published
revisions and activation are managed independently, Splunk validates the SPL before every preview
and run, and each policy’s runs are in Collection health. See Findings, Finding groups and findings actions.
Collection jobs¶
Job |
Schedule |
What it does |
|---|---|---|
Inventory ( |
Every 10 minutes |
The Splunk instances of the host scope (roles and GUIDs; a universal forwarder only when a group declares it) and what they forwarded, indexed and ran over a day; with what the registrations job published, the directory: the collectors, the utility nodes, their inputs and groups. Rebuilt about hourly, sooner when the registrations moved, always on a manual run or after a saved setting. |
Registrations ( |
Every 10 minutes |
What splunkd registered (input commands, modular input schemes) and the starts, forward from its index-time cursor; a new collector back to its last start (up to 30 days), a slice at a time, carried over. Most runs send no search. Until every collector is read, the enablement decisions hold. |
Live ( |
Every 5 minutes |
The index-time window since the previous run (with catch-up): what each monitored collector
took in and forwarded. Assembles the latest facts of the jobs below into the snapshot, writes
the series, then evaluates every policy in the same search. Never builds the directory: a run
before the first inventory waits for it ( |
Pipeline ( |
Every 5 minutes |
The queues, the output trouble, and what each collector sent per output group and destination. |
Resources ( |
Every 5 minutes |
CPU and memory; the disks over the last 30 minutes. |
Inputs ( |
Every 5 minutes |
The running input processes and the lines they wrote, by the level each line states itself. |
Add-on signals ( |
Every 5 minutes |
The lines and error lines of each add-on log file, what each source delivered; on syslog or file collectors, what reached the indexes from their monitor stanzas. |
Changes ( |
Every 5 minutes |
The configuration changes, the parsing quality (the last 15 minutes) and the file backlog. |
Collector trouble ( |
Every 5 minutes |
The health report flags and indicators, the collectors’ own trouble lines, the files of the flagged collectors only; the day-long read once an hour, read last. |
Destinations ( |
Every 5 minutes, just before the live run |
The destinations the senders fail to reach, their writers, and the cluster peers restarting. |
Enablement ( |
Every 15 minutes |
The configuration changes of the files defining inputs, replayed in time order (a new collector from 90 days back, a week at a time), then the optional API accounts. |
Traffic ML models ( |
Twice an hour |
Learns each monitored collector’s collected volume per hour of the week over the Model learning history, then the new buckets every 6 hours. |
Policies ( |
On demand |
Evaluates the policies on the latest live publication. |
Twelve reports are scheduled per tenant (the Policies report runs on demand): the most of any
product, so a Splunk Cloud stack’s search concurrency applies first here. Schedules are offset per
tenant. Collect now (the status strip) runs the live job; Run now in Collection health runs a
job; Scheduled collection (Settings → Collection) is on by default. The reports are
trackme_splunk_collector_<job>_<generation>_tenant_<tenant>, and the AI Agent report
trackme_fwc_agent_tracker_tenant_<tenant> (scheduled only once the scheduled triage is enabled).
Each job has its own report, time budget (its interval less 30 seconds), lock, receipt and Collection
health line, and a search may use what is left of its job’s budget: there is no fixed per-search
timeout and no row limit. The fact jobs run in the minutes between two live runs; a job that misses a
slot makes the policies hold, never count twice, and facts older than three runs of their job read as
not read. Every read about the collectors names them, so a universal forwarder fleet never adds to its
cost. With Scheduled collection off, a Collect now starts the other jobs beside the live run,
and it takes a second Collect now before the policies judge anything. The reports run
| trackmesplunkcollect (log trackme_splunk_collect.log); several jobs of a tenant can overlap
under the tenant owner, so a least-privilege owner may need a higher search quota.
Settings reference¶
Settings is one browser with a single Save settings. A tenant role that is not a TrackMe administrator of the tenant sees it read-only. Changing a field marked starts a new source restarts collection, and the findings, run history and traffic models of the previous source start again. Every policy threshold is a policy setting; the expected collectors are set in their policy and the inputs that must stay enabled in Inventory → Inputs.
Section |
Setting |
Default |
Notes |
|---|---|---|---|
Source |
Source (search tier) |
|
Fixed once the tenant exists: create another tenant for another deployment. |
Internal, Metrics, Introspection index |
|
Fixed once the tenant exists; an empty Metrics index means |
|
Telemetry form |
Automatic |
Automatic, |
|
Host patterns |
|
The Splunk instances that may be collectors, by host name ( |
|
Excluded hosts |
None |
Never |
|
Utility nodes |
Left out |
Licence managers, monitoring consoles and other full instances that forward without collecting: listed apart, monitored only when included; starts a new source. |
|
Repair workspace |
— |
Recreates the tenant’s collections and reports; findings and settings are kept. |
|
Groups |
Groups |
None |
Up to 200. Each: a name (1 to 80 characters), host patterns (up to 50), expected inputs (up to
100 patterns over a command line, a modular input scheme or a |
Syslog or file collectors |
Off |
Declares the group’s members as forwarders collecting other machines’ data from files (rsyslog or syslog-ng writing to disk): TrackMe never detects them. Its host patterns need a literal part. A universal forwarder is monitored only in such a group. Add collector (Settings → Groups, Inventory) declares host names or patterns into a group. |
|
New collectors join the groups on their own |
On |
Off: only the assigned collectors belong to a group. |
|
Assigned collectors |
None |
Up to 500 host names, each to one group, which it belongs to first whatever the patterns say. |
|
Inputs |
Excluded inputs |
Splunk’s own inputs |
Hidden from the views and never evidence of collection: up to 100 patterns over a command line or a modular input scheme; empty shows every input. Restore the default exclusions brings the default list back. |
API |
Collector API accounts |
None |
Optional, up to 200 remote accounts on the collectors themselves (never |
Collection |
Scheduled collection |
On |
|
Model learning history |
14 days |
0, 7, 14, 30, 60 or 90 days. |
|
Traffic ML models |
Model defaults |
— |
Sensitivity, smallest deviation, seasonality, recency of every collector’s model. |
Notifications, AI Agent, Assignment, Finding actions |
Shared settings of the findings tenants |
Off / none |
See Findings, Finding groups and findings actions; the assignment criterion is the collector’s group. |
Groups, assignments, the input exclusions and the API accounts apply from the next collection and never start a new source.
Input states¶
Monitoring → Inputs lists each input as configured, per collector; Processes lists what runs. The state is the strongest evidence of the window:
State |
Means |
|---|---|
Disabled |
Its configuration says so (an API account when it lists the input, else its last configuration change). |
Delivering |
Its own source took data in, or, on a syslog or file collector, its monitor stanza’s files reached the indexes. |
Logging |
Its own log file took lines. |
Running, Writing |
Its process was sampled, or wrote to stderr: on an input, only when it is the only input of that process. |
Its process runs |
A process ran: a process names no input, so this says nothing more of this one. |
No activity evidence |
Nothing in this window. Never a failure on its own: a daily input between two runs reads this way. |
Collection health codes¶
The Collection health tab and the Configuration Guardian check cribl_collection_health
(warning) report, while Scheduled collection is on:
Code |
Severity |
Meaning |
|---|---|---|
|
Warning, Failure, Failure |
No live run 20 minutes after configuration; none for 20 minutes; runs not completing for 30 minutes. |
|
Warning, Failure, Failure |
No inventory run; none for 3 hours; not completing for 90 minutes. |
|
Warning, Failure, Failure |
A fact job missed three of its runs (15 minutes for a five-minute job, 30 for Registrations, 45 for Enablement): never ran since the first assembled snapshot, or none since; its runs not completing for 30 minutes. Until its next run, the policies reading its facts hold. |
|
Failure |
Policy evaluation not completing for 30 minutes. |
|
Failure |
Traffic model training not completing for 90 minutes: Collected volume outlier reads the last expected ranges, then its collectors read as learning. |
|
Warning |
An hour after configuration, the source shows no Splunk instance in the host scope writing its metrics. |
|
Warning |
The last inventory found no collector running an input of its own: the instances of the scope are utility nodes, or run only Splunk’s own inputs. |
|
Warning |
An API account could not read the inputs of its collector: the telemetry speaks for it meanwhile. |
|
Warning |
Notification emails failing. |
|
Warning |
Custom policies: schedule drift, failing runs, missed schedules, recovery required. |
A run also states its reason on its receipt (Collection health → Trackers), such as
fwc_directory_pending (waiting for the first directory), fwc_live_stale (the live window is
more than 20 minutes old: the policies judge nothing) or live_read_failed (one read of the live
run failed: its facts are missing from this window).
AI Agent¶
The Splunk Forwarder Collectors AI Agent settings (Settings → AI Agent, or Manage AI Agents):
Setting |
Default |
Notes |
|---|---|---|
Scheduled triage |
Off |
Hourly when on ( |
Mode |
Inspect |
Inspect writes assessments and notes; act also applies the decisions allowed below. |
AI provider |
First available |
|
Findings per run, time budget per run |
10, 3,000 s |
|
Review interval |
1 day |
A finding reviewed by a scheduled run is skipped that long unless it moved. |
Allow investigating, allow temporary dismissal, allow resolve |
Yes, yes (up to 7 days), no |
A scheduled run never overrides an operator decision. It never enables, disables or restarts an input, never forgets a collector and never adds one to the expected collectors. |
Custom instructions |
None |
Added to the agent’s instructions; Evaluate… tries them on a finding. |
An input that must stay enabled and is still disabled or absent is a real problem: TrackMe corrects a
review that says otherwise before it is shown, and says so in its summary. The agent cannot dismiss
that finding while it fails. Runs are audited with the sourcetypes
trackme:ai_agent:fwc_agent:inspect and trackme:ai_agent:fwc_agent:act. See
AI Agents for findings.
Indexed finding events¶
Every finding transition is written to the tenant’s notable index as sourcetype=
trackme:cribl:finding (and Finding groups as trackme:cribl:finding_group), with
object_category=splunk_collector, whatever the notification settings. The tenant’s series are in
its metric index under trackme.fwc.host, trackme.fwc.input, trackme.fwc.group and
trackme.fwc.fleet. The lookup trackme_splunk_collector_collectors_tenant_<tenant> holds one
record per collector and utility node of the current directory, for | inputlookup or | lookup
in your own searches.
Permissions¶
Any role of the tenant reads the workspace.
The tenant’s power and admin roles triage findings, stop and resume the monitoring of a collector, and Forget collectors.
A TrackMe administrator of the tenant (
trackmeadminoperationsand an admin role of the tenant) edits Settings, policies, groups, custom policies, the expected collectors (Add to expected is a policy save) and the inputs that must stay enabled (Inventory → Inputs, a policy save too).The scheduled runs search as the tenant owner: every remote account (the source, the API accounts) and lookup table the tenant reads must be usable by the owner, and on syslog or file collectors the owner needs search access to the indexes their files go to (without it, a stanza stays No activity evidence).
Commands¶
Command |
What it does |
|---|---|
|
The wizard’s discovery (read-only, nothing saved): two searches of five minutes each, the
instances of the host scope over the hour of their instance line, then what they forwarded,
indexed and ran over the last 15 minutes. A preview, never the tenant’s list. With
|
|
TrackMe’s series of collectors, inputs, a group or the fleet ( |
|
The Last 5 minutes figures of the tenant, a group or a collector ( |
|
The raw lines behind a figure of a collector or a group’s members ( |