Splunk HEC reference

The reference of the Splunk HEC tenant: the built-in policies and their defaults, the HEC reply codes, the collection jobs, the limits, the history TrackMe keeps, the settings, the AI Agent, the indexed finding events, the permissions, the Configuration Guardian conditions, the run reasons and the REST endpoints. The user guide is Splunk HEC; the findings model shared with User Activity Monitoring, Cribl Stream and Cribl Edge is Findings, Finding groups and findings actions.

Built-in policies

A Splunk HEC tenant has eighteen built-in policies. Seventeen are on out of the box; Endpoint resources is off (it watches the host, not HEC). Token silence is on but decides nothing until tokens are selected, and Token consistency decides only with the optional splunkd API. Your own SPL policies on endpoints, pools and tokens are described in Custom policies.

Endpoints

Policy	What it detects	Default	Severity
Endpoint availability	An endpoint seen recently sends no HEC heartbeat.	On	High → Critical
Senders refused	Requests refused as server busy (replies 9, 18–20, 23, 26, 27): HEC could not queue them.	On	High → Critical
Back pressure	An ingestion queue of the endpoint backed up; the finding names the stage that could not keep up and whether its thread was busy.	On	Medium → High
Output blocked	The endpoint’s output to its receivers is the bottleneck, paused, or failing to connect.	On	High
Error rate	Refused requests out of the requests received (empty load-balancer probes excluded).	On	Medium
Authentication failures	Requests refused for an invalid, disabled or missing token.	On	Medium
ACK pressure	Indexer acknowledgment under pressure: the ACK store full, or replies saying the ACK service is unavailable, above its warning level or at capacity.	On	High
ACK not polled	Requests are acknowledged but no sender polls their acknowledgments.	On	Medium
Endpoint resources	CPU or memory of the endpoint’s host at or above the threshold.	Off	Medium

Pools

Policy	What it detects	Default	Severity
Pool capacity	Fewer pool members up than expected.	On	High → Critical
Pool imbalance	A pool member up but receiving a very small share of the pool’s events.	On	Low
Pool traffic drop	The pool receives far fewer events than it usually does at this hour of the day.	On	High → Critical
Pool traffic outlier	The pool’s events (or bytes) outside what its traffic model expects at this hour of the week.	On	Medium
Token consistency	A token configured on some pool members only, or with a different status, index, allowed indexes or ACK from one member to another (splunkd API).	On	High
Configuration drift	Pool members running different Splunk versions or, with the splunkd API, different HEC settings, for longer than the grace period.	On	Medium

Tokens

Policy	What it detects	Default	Severity
Rejected data	Requests of a token refused for their content, or events dropped as oversized.	On	Medium → High
Token silence	A selected token sent no events for longer than the silence period.	On	High
Token traffic outlier	A token’s events (or bytes) outside what its traffic model expects at this hour of the week.	On	Medium

Default settings

What each policy applies to, its checks to open and to recover (one check per collected 5-minute window), and its own settings with their defaults and ranges.

Policy (id)	One finding per	Open / recover	Settings and notes (defaults)
Endpoint availability (hec_endpoint_availability)	Endpoint	1 / 2	Silent after 300 s (300–86,400): no heartbeat for that long. Dormant after 7 days (1, 7 or 30): a longer silence opens nothing. Replacement grace 1,800 s (300–86,400): in a pool of replaceable members, a silent member waits that long for a replacement (see Splunk HEC, Pools). A late endpoint (a heartbeat in the last 5 minutes) and a stalled one (its output blocked) are held. Critical when every member of the pool is down.
Senders refused (hec_server_busy)	Endpoint	2 / 3	Open at 10 busy refusals in a window (1–1,000,000,000). Critical at 20 % of the requests (1–100). Busy replies 9, 18–20, 23, 26, 27, and the HEC queue insertion failures when the endpoint reports them. Decided for an endpoint up in the window.
Back pressure (hec_queue_saturation)	Endpoint	3 / 3	No setting. A queue of the HEC path is saturated when its 90th percentile fill reaches 80 % or it was reported blocked in 30 % of its samples or more; back pressure is a saturated queue or busy refusals. High when the endpoint also refuses requests as busy.
Output blocked (hec_forwarding_blocked)	Endpoint	2 / 2	Pause counted from 30 s (1–3,600): an output pause that long in the window; a failing destination or the output as the bottleneck counts too, and so does a stalled endpoint.
Error rate (hec_error_rate)	Endpoint	3 / 3	Open at 5 % refused (0.01–100), Recover below 2 % (below the open threshold), At least 100 requests in the window (1–1,000,000,000). Load balancer probes (reply 5) excluded.
Authentication failures (hec_auth_failures)	Endpoint	2 / 3	Open at 20 token refusals in a window (1–1,000,000,000): replies 1, 2, 3, 4, 21, 22. The detail names the reply codes and the clients refused most.
ACK pressure (hec_ack_pressure)	Endpoint	3 / 3	No setting. The ACK service logged its store full (HttpInputAckService), or requests were refused with replies 19, 20, 25 or 27.
ACK not polled (hec_ack_unpolled)	Endpoint	6 / 2	Open at 1 request acknowledged (1–1,000,000,000) in a window that received no ACK poll; after 6 checks (30 minutes), so a sender polling every few minutes never opens it. Decided where Splunk reports both counters (9.4 and 10.x); on 10.x the detail names the tokens acknowledged with no poll.
Endpoint resources (hec_endpoint_resources)	Endpoint	3 / 3	Off by default. Open at 90 % CPU or memory (0–100), Recover at 80 % (below the open threshold).
Pool capacity (hec_pool_capacity)	Pool	2 / 2	Critical below 50 % of the expected members (1–100), or with none up. Expected size: the pool’s own, else learned (the second highest daily peak of members up over 7 days). Nothing is decided while the size is being learned.
Pool imbalance (hec_pool_imbalance)	Pool	6 / 3	Smallest share 5 % of the pool’s events (0.01–50), At least 10,000 events in the window (1–1,000,000,000,000), at least two members up. A member first seen less than 15 minutes ago is skipped (warm-up).
Pool traffic drop (hec_pool_traffic_drop)	Pool	3 / 2	Open below 10 % of the usual traffic (1–90): the median rate of the same hour of the day over the last 7 days, from 3 days at least. At least 1,000 usual events an hour (1–1,000,000,000,000). Critical when the pool received no event. Holds on bank holidays by default.
Pool traffic outlier (hec_pool_traffic_outlier)	Pool	2 / 2	Traffic judged: events (default) or bytes received. Direction: drops and spikes (default), drops only, spikes only. At least 100 per five minutes (0 or more): below it in both the window and the model, nothing is judged. Open from: normal confidence (7 days learned, default) or low confidence (1 day). Holds on bank holidays by default.
Token consistency (hec_token_consistency)	Pool	2 / 2	No setting. Compares the members up of a pool with fresh splunkd API facts (two at least, else nothing is decided): a token missing on some members (only when no member’s token list was cut at 200), or differing in status, index, allowed indexes or ACK.
Configuration drift (hec_config_drift)	Pool	2 / 2	Grace 60 minutes (5–10,080): a difference lasting that long opens a finding. Splunk versions (from the telemetry) and, with fresh splunkd API facts on two members at least, the global [http] settings. Splunk Cloud members are never compared.
Rejected data (hec_rejected_data)	Token	2 / 3	Open at 10 rejected requests in a window (1–1,000,000,000): replies 6, 7, 12, 13, 15; any oversized event opens too. High for an incorrect index (reply 7) or oversized events (lost events).
Token silence (hec_token_silence)	Token	2 / 2	Silent after 900 s (300–86,400). Tokens to watch: up to 100 typed names or patterns and up to 20 lookup tables (see Token silence selection). Nothing is decided until tokens are selected. Holds on bank holidays by default.
Token traffic outlier (hec_token_traffic_outlier)	Token	2 / 2	The settings of Pool traffic outlier. A modelled token absent from the window is judged at zero. Holds on bank holidays by default.

Settings every policy has. On / off; severity (Low, Medium, High, Critical; the policies above raise it in the cases listed); checks to open and checks to recover (1 to 12 consecutive windows, one every 5 minutes); During bank holidays (Hold new findings or Keep opening findings; see Bank holidays); and for the endpoint and pool policies, a pool scope (empty: every pool; the endpoint policies also take the discovered endpoints). A token policy has no pool scope.

How a check works. A check is one collected 5-minute window: the policies run after every HEC activity run, on the window it published. A window older than 20 minutes is not judged (hec_live_stale). The counter policies judge only the endpoints measured in the window.

How findings close by themselves. A finding recovers when the measured condition clears for the checks to recover. It closes automatically when its object leaves the tenant’s scope (a host pattern or pool changed, a policy turned off or out of scope), when its endpoint was replaced in a replaceable pool (closing reason endpoint replaced, never a recovery), or when nothing evaluated it for 24 hours. A condition TrackMe cannot decide (unknown, for example a pool still learning its size) is never written as a recovery.

HEC reply codes

The reply code Splunk logs for each refused request (reply= in HttpInputDataHandler), and how the tenant counts it:

Class	Reply codes	Counted by
Busy	9, 18, 19, 20, 23, 26, 27	Senders refused, Back pressure, Error rate
Token refused	1, 2, 3, 4, 21, 22	Authentication failures, Error rate
Rejected data	6, 7, 12, 13, 15	Rejected data, Error rate
ACK	19, 20, 25, 27	ACK pressure
Channel, configuration	10, 11, 14, 16	Error rate
No data (load balancer probes)	5	Counted and shown, never a sender error
Capacity warning	24, 25	Shown (only with Splunk’s warn_at_80 back pressure state)

num_of_errors of the HEC metrics is collected but never used: it does not count refusals. The error rate is the refused requests (every class but the probes) over num_of_requests, which includes the refused requests.

Bank holidays

A Splunk HEC tenant uses the bank holiday calendars of its scope (Bank holidays). While a holiday is active, the policies set to hold open no new finding; a finding already open is still evaluated and can recover and close.

Policy	Default
Pool traffic drop, Pool traffic outlier, Token traffic outlier, Token silence	Hold new findings
Every other policy	Keep opening findings

A rule saved before this setting existed reads its default. When the calendars cannot be read, nothing is held. The traffic models leave the hours of the tenant’s holidays out of their training when the Exclude from ML training effect applies to the tenant; a run that cannot read the holidays learns nothing and fails with ml_holidays_unavailable (the next run retries).

Custom policies

A custom policy is an SPL search on the tenant’s Splunk data whose result rows each name a HEC object. How to write one: Splunk HEC, section Custom policies.

Object type	Resolution
Endpoint	A host name within the tenant’s host patterns; its pool is the one the configuration assigns (discovered when none).
Pool	A pool id of the tenant.
Token	A token name.

• A custom finding takes the subject of the built-in findings on the same object, so it joins that object’s Finding group.

• A host outside the tenant’s host patterns or a pool the tenant does not define is not monitored: skipped, counted in the run’s receipt, and its earlier findings recover. Only a value that cannot name an object is unresolved; a run with an unresolved row recovers nothing.

• Runs, lifecycle and limits: the same as on a Cribl Stream tenant, see Custom policies.

• Health: Collection health → Custom policies, and the custom_* problems of the Configuration Guardian check cribl_collection_health.

Collection jobs

Each Splunk HEC tenant has its collection reports, named trackme_splunk_hec_<job>_<generation>_tenant_<tenant_id>, run as the tenant owner, and the AI Agent report (below). Each tenant’s schedule runs at a minute offset of its own.

Job (Collection health label)	Schedule	What it does
live (HEC activity)	Every 5 minutes	Reads the last settled 5-minute window by index time, in two waves of searches dispatched together (wave 1: the instance and token series of the HEC metrics; wave 2: the refused requests, the queues and thread duty cycles, the output, the host resources), then the receivers of the endpoints whose output was in trouble. Publishes the window, writes the history points, evaluates the policies, sends the notifications.
inventory (Endpoint directory)	Every 10 minutes; rebuilt about once an hour	Rebuilds the endpoint and token directory (the endpoints seen in the last 24 hours, kept while they reported in the last 7 days; the token names seen in the last 24 hours; the identity from Hostwide) and, with accounts bound, reads the splunkd API. Between rebuilds, confirms the directory is current. A manual run always rebuilds.
mltrain (Traffic models)	Twice an hour	Learns the traffic models of the pools and tokens from the HEC metrics, then projects the expected ranges of the next 24 hours.
policies (Monitoring policies)	After every live run; on demand	Evaluates the policies on the newest window (Run now).
trackme_hec_agent_tracker_tenant_<tenant_id> (AI Agent)	Hourly, when enabled	Scheduled triage by the Splunk HEC AI Agent. Off by default: the report stays unscheduled until the triage is turned on.

Run budgets. Each job runs at most its interval less 30 seconds (the live and policies jobs 270 s, the directory 570 s, the traffic models 1,770 s), and a scheduled run ends 30 seconds before its next scheduled run, however late the scheduler started it: a late run reads less and the next one carries on. A run started with less than 15 seconds left does nothing and says so (collection_time_budget). The tenant’s reports carry no scheduler skew (allow_skew = 0, set again at every schedule reconciliation).

Windows. A live run reads what became searchable since the previous run, up to the last 5-minute boundary at least 120 seconds old (the settle delay, never less than 60 seconds), and scans 30 minutes of event time to find it: telemetry more than 30 minutes late is not counted. After missed runs, at most three windows are read (one history point each); older windows stay gaps. A window already counted is never read again. Rates are per second of measured activity.

Searches. A search is judged by its job state: Splunk messages attached to a search are recorded on the run (never a failure), and a search Splunk completed short is used and flagged. Only a failed job, a time limit or a broken read fails the run. The searches name the directory’s endpoints (up to 500, else the host patterns).

Scheduled collection off (Settings → Collection) unschedules the reports; Collect now and Run now still work. A new tenant’s first live run builds the directory itself.

Limits and history

Bound	Value
Endpoints per run	5,000 (beyond, counted as truncated; Collection health says so)
Tokens per run	20,000
Refused-request rows, output rows, receiver rows per run	20,000 each
Queue rows per run	50,000
Receivers read per run	200
Client addresses kept per endpoint	20
Indexes refused kept per token	10
Endpoints listed per token	200
Pools per tenant	200 (host patterns per pool: 50)
Tenant host patterns	100
Hand assignments	500
Expected size of a pool	1 to 10,000
splunkd API accounts / URLs per account / tokens read per endpoint	20 / 50 / 200
Token silence: typed names / lookup tables / names per lookup	100 / 20 / 100,000
Token silence: tokens selected	20,000; at most 250 new silence findings started per run (the others start at the next runs)
Token traffic models	500 tokens (busiest first)
Traffic model history learned on the first runs (Model learning history)	14 days by default; from today on, 7, 14, 30, 60 or 90 days (at most what _introspection keeps: 14 days by default)
Traffic models with their own settings	500
Endpoints, pools or tokens in one Topology Studio group	2,000 per kind (narrow the group beyond)
HEC references per tenant in one Topology Studio view	100 (use groups for more)
Rows per page of the Endpoints and Tokens tabs	Up to 100 (the viewer’s preference)

History kept by TrackMe. At every live run, in the tenant’s TrackMe metric index (through trackme_hec_metrics.log):

• trackme.hec.endpoint.* per endpoint (dimension endpoint, the host): up, requests, events, bytes_received, errors, parser_errors, auth_failures, busy_replies, refused, failures, probe_noise, waiting_ack, oversized_events, worst_queue_fill, queue_insertion_failures, back_pressure, output_paused_seconds, cpu_pct, mem_used_pct;

• trackme.hec.pool.* per pool (dimension pool_id): members, up, down, stalled, requests, events, bytes_received, auth_failures, busy_replies, refused, failures, oversized_events, waiting_ack, worst_queue_fill, back_pressure;

• trackme.hec.token.* (dimension token_id, a digest of the name, never the name): events, bytes_received, requests, refused, for the 50 busiest tokens of the window with 10 places kept for tokens with refused requests.

Every point carries tenant_id and the end of its window. They draw the trends of the workspace, the drilldown histories and the Topology Studio sparklines. waiting_ack is the ACK-enabled requests of the busiest minute, a count per interval, not a backlog.

Settings reference

Setting (where)	Default	Meaning
Source (wizard; Settings → Source & pools)	local	This search tier, or a TrackMe remote account to the search tier that sees the endpoints’ internal indexes. The tenant owner must hold one of the account’s roles. Locked once configured.
Internal index, Introspection index (same)	_internal, _introspection	Where the endpoints’ splunkd.log / metrics.log and HEC metrics are. Locked once configured.
Endpoint host patterns, Excluded hosts (same)	*, none	The endpoints in scope (* and ? wildcards). Changing them starts a new source revision; an excluded host is never monitored.
Pools (same)	A default pool	Identifier (lowercase letters, digits, - and _, up to 40 characters, fixed once saved; never discovered or tokens), name (up to 80 characters), host patterns (the default pool needs none), expected size, replaceable.
Assignments (same, the endpoint’s pool pill)	None	An endpoint pinned to a pool by its host name; never to a replaceable pool.
Auto-join (same)	On	A new endpoint no pool claims joins the default pool; off, it waits as discovered.
splunkd API accounts (same)	None	Up to 20 remote accounts (never local) whose splunkd the directory reads. See Splunk HEC, The splunkd API enrichment.
Scheduled collection (Settings → Collection)	On	Runs the live job every 5 minutes, the directory about hourly, the traffic models twice an hour.
Model learning history (wizard; Settings → Collection)	14 days	How far back the traffic models learn on the first runs, and what a relearned model reads: from today on, 7, 14, 30, 60 or 90 days.
Traffic ML models (Settings → Traffic ML models; a model’s Tune model)	Recent days half the weight after 14 days, automatic seasonality, weekday pattern after 28 days, findings once 7 days are learned, normal sensitivity, deviations under 10 % ignored	The defaults every traffic model follows; a model’s own settings add the training days, the direction, a pause and excluded periods. See Splunk HEC, Traffic models.
Bank holidays (Manage Bank Holidays; Configure tenant → Bank Holidays)	The system default calendars	The tenant’s calendars. See Bank holidays.
Notifications (Settings → Notifications)	Off	Finding emails: events, minimum severity, policies, delivery account and recipients, digest, charts.
Assignment (Settings → Assignment)	None	Assignment groups, team email, ordered routing rules, manual overrides. See Assignment and automatic routing.
Finding actions (Settings → Finding actions; a policy’s Actions and Automations)	None	Drilldown, link and webhook actions, and the Automations that call them. See Findings actions.
AI Agent (Settings → AI Agent)	Scheduled triage off	See AI Agent.
Policies (Policies tab)	See Built-in policies	Saved for the tenant with their own revision.

The span of a window (300 seconds) and the settle delay (120 seconds, up to 600) are part of the source configuration. The tenant’s indexes, roles and owner are set in the wizard and edited like any tenant’s.

Token silence selection

Token silence watches the tokens that must keep sending, from two sources that add up:

Source	Rules
Typed token names	Up to 100 names or patterns, case-sensitive (a token name is the HEC input’s name), 1 to 256 characters without a control character, a quote or a backslash. * matches any characters and ? one character.
Splunk lookup tables	Up to 20 per policy, each a lookup shared with the TrackMe app, on the local search tier or on a remote deployment through a TrackMe remote account (never a TrackMe KV store lookup), with the field holding the token names. Up to 100,000 distinct names per lookup; a larger one is refused (too_large), never truncated. Values are trimmed; values that cannot be a token name are counted as ignored.
Match mode (per lookup)	Exact (default): whole token names; a value holding * or ? is ignored. Wildcard: * and ? match.

Decisions. Every name selected exactly — typed or from an Exact lookup — is selected whether seen or not: a name not seen sending in the last 24 hours is silent at once. The patterns select the known names they match (the latest window, the 24-hour directory, the tokens already followed). At most 20,000 tokens are selected, exact names first.

Refresh. The policies run reads each lookup again every 15 minutes, the oldest first, and keeps its names between reads. A lookup that cannot be read keeps its last good names; a lookup never read successfully selects nothing. Collection health reports token_lookups_failing when a lookup is unreadable, too large or empty.

Test selection runs as a search: it reads every lookup now and matches the published token directory and the latest window, then shows the tokens selected by state (sending within the draft’s silence period, silent, not seen in 24 hours), up to 50 examples (not seen first) with the entry that selected each, each source’s names, and the patterns selecting none. It saves nothing. Token names are selectors: they are not copied into the evidence of the findings.

AI Agent

The Splunk HEC AI Agent triages the tenant’s findings from what TrackMe holds: the finding and its HEC detail, history, episodes and Finding group; its endpoint, pool or token in the published directory and latest window; whether the endpoint or pool is still there; Collection health; and its own past decisions. It runs no search. The user guide is in Splunk HEC; the shared model in AI Agents for findings.

Assessments: true positive, likely false positive, expected behaviour, endpoint or pool not in use, needs investigation.

Settings (Settings → AI Agent; the core ones also in Manage AI Agents automation, AI Agents automation, and in Configure tenant), stored in the tenant record:

Setting	Field	Default	Range and notes
Scheduled triage	ai_hec_agent_enabled	Off (0)	Asking the agent needs only AI to be available.
Mode	ai_hec_agent_mode	Inspect	inspect (recommend only) or act (apply allowed decisions); scheduled runs.
AI provider	ai_hec_agent_provider_name	First available	An AI provider account name.
Review interval (days)	ai_hec_agent_min_days_between_reviews	1	0 reviews every run; a new episode is reviewed at once.
Findings per run	ai_hec_agent_max_findings_per_run	10	1 to 50, most severe first.
Time budget per run (seconds)	ai_hec_agent_max_runtime_sec	3,000	60 to 86,400.
Custom instructions	ai_hec_agent_custom_instructions	Empty	At most 4,000 characters, line breaks kept; followed by every run; never widen what the agent may do.
Allow Mark investigating	ai_hec_agent_allow_investigating	Allowed (1)	An open finding only.
Allow Dismiss for a time	ai_hec_agent_allow_dismiss_temporary	Allowed (1)	A critical or high finding still failing only when its endpoint is no longer in the directory (retired, dormant or replaced) or its pool is no longer defined.
Longest dismissal (days)	ai_hec_agent_dismiss_max_days	7	1 to 90.
Allow Resolve	ai_hec_agent_allow_resolve	Not allowed (0)	Only after a measured recovery.

The shared filters of the automated AI reviews, ai_automated_priority_filter (by default critical and high findings) and ai_automated_filter_expression, apply to the scheduled selection with the finding severity as the priority.

Gone from the directory. An endpoint is gone only when a current directory no longer lists it and the finding has not been evaluated for an hour; a pool is gone when the tenant no longer defines it. A token is never gone.

Scheduled triage runs the report trackme_hec_agent_tracker_tenant_<tenant_id> hourly when all of these hold: the AI Assistant is enabled, an AI provider is configured, the tenant is enabled, and scheduled triage is on. The report runs as the tenant owner, which must be a Splunk administrator account (a member of admin or sc_admin), not nobody: the run records itself in a store only they can write. Saving the settings schedules or unschedules it, and the tenant health tracker checks the same conditions every five minutes.

Audit. Every run is indexed to the tenant summary index with the sourcetype trackme:ai_agent:hec_agent:inspect or trackme:ai_agent:hec_agent:act; decisions appear in the finding’s history under the actor AI Agent, and the reviewed finding carries one current [AI Agent] note.

index=trackme_summary sourcetype="trackme:ai_agent:hec_agent:*" tenant_id="example-hec"
| table _time, sourcetype, launched_by, automated, status

Indexed finding events

Every transition of a finding and of a Finding group is written as an event, whatever the notification settings:

Item	Value
Index	The tenant’s notable index (trackme_notable by default)
Sourcetypes	trackme:cribl:finding (a finding) and trackme:cribl:finding_group (a Finding group), shared with the Cribl tenants
object_category	splunk_hec (with tenant_id)

Example: the HEC findings opened or reopened in the last day.

index=trackme_notable sourcetype=trackme:cribl:finding object_category=splunk_hec tenant_id="example-hec"
    transition IN ("opened", "reopened")
| table _time, policy_name, entity_name, severity, reason, finding_id

Permissions

Each tier needs the TrackMe capability and one of the tenant’s matching roles; a Splunk administrator (admin, sc_admin) holds every tier on every tenant.

Action	Who
Read every tab, the drilldowns, the last five minutes and Why it was flagged; run drilldown and link actions	Any role of the tenant (trackmeuseroperations). The reads run with your own rights: a remote source also checks the remote account’s roles.
Triage findings and Finding groups (investigate, assign, snooze, dismiss, resolve, change the severity, group by hand), notes; ask the AI Agent and let it act on its inspection	The tenant’s power and admin roles (trackmepoweroperations)
Policies (on / off, severity, conditions, scope, bank holidays), the Token silence lookups and Test selection, custom policies	A TrackMe administrator of the tenant (trackmeadminoperations and a tenant admin role)
Source & pools, Collection, Collect now / Run now, Repair workspace, Notifications, AI Agent settings, Assignment settings, Finding actions and Automations	A TrackMe administrator of the tenant
Create or delete the tenant; manage remote accounts and Webhook accounts	TrackMe administrators (see Remote Splunk deployments and Webhook accounts)

The scheduled AI Agent needs a tenant owner that is a Splunk administrator account (not nobody). A remote account used as the source or for the splunkd API must be usable by the tenant owner (one of its roles).

Configuration Guardian

The tenant check cribl_collection_health raises Splunk HEC collection needs attention (warning) when collection does not run as configured, and clears by itself on the first run without a problem. The Collection health tab shows the same conditions.

Condition	In plain words
live_never_ran, live_not_running	No live HEC run since the source was configured, or none for 20 minutes. Check that the reports exist, are scheduled and that their owner can run searches.
live_failing, inventory_failing, policies_failing	A job keeps failing (30 minutes for live and policies, 90 minutes for the directory). Search the collector log, and check that the tenant’s remote account connects.
inventory_never_ran, inventory_not_running	No endpoint directory since the source was configured, or none for 3 hours.
no_hec_endpoints	No endpoint for an hour after configuration: HEC disabled on the endpoints, the internal and introspection indexes not searchable from the source, or host patterns matching nothing.
capacity_reached	A collection limit was reached (more endpoints or rows than a run keeps). Narrow the host patterns or exclude hosts.
token_lookups_failing	A Token silence lookup table could not be read, holds more than 100,000 names, or is empty, at the last policies run. A lookup that cannot be read keeps its last names. Check the lookup, its sharing with the TrackMe app, and the remote account when it is on a remote deployment.
custom_schedule_drift, custom_policy_failing, custom_missed_schedule, custom_recovery_required, custom_health_unavailable	A custom policy’s report or runs need attention: the same conditions as on a Cribl Stream tenant (see Custom policies).
notification_delivery_failing	Notifications are on and the emails have failed for 30 minutes. Collection never waits for delivery.

Each alert carries its recommended actions. See Configuration Guardian. Also relevant: ai_automation_owner_is_system_user warns when the AI Agent’s scheduled triage is on and the tenant owner is nobody or a system account.

Run reasons

The outcome each job records, shown on its Collection health card:

Reason	Meaning
live_published / window_current	The latest HEC window was collected / was already collected.
directory_published / directory_current	The endpoint directory was rebuilt / is up to date.
hec_policies_evaluated	The policies were evaluated on the latest window.
hec_live_not_published	No HEC window published yet.
hec_live_stale	The latest window is more than 20 minutes old: it is not judged.
hec_policy_time_budget	The policies ran out of time.
hec_scope_changed, source_revision_changed	The source changed during the run; the next run reads the current settings.
hec_snapshot_changed	A new window was published while reading; the next run evaluates it.
hec_policies_unavailable	The policies could not be evaluated.
remote_account_refused	The remote account refused the tenant owner. Add one of the owner’s roles to the account, or change the owner.
search_failed, search_timeout	A collection search failed or exceeded its time limit. The collector log carries the Splunk messages of the failed search.
models_current, training_continues	The traffic models are up to date / are still learning the history (the next run continues).
ml_holidays_unavailable	The tenant’s bank holidays could not be read: the training waited.
job_lock_busy	Another run of the same job was running; logged as information.

The collector log is index=_internal sourcetype=trackme:custom_commands:trackmecriblcollect (file trackme_cribl_collect.log).

REST endpoints

The workspace uses the /trackme/v2/cribl/tenant/* endpoints shared with the Cribl tenants (workspace, configure, collect, collect_result, repair, policies, policy_save, findings, finding, triage, finding_groups, group_triage, group_members, notifications, notification_test, the custom policy and AI Agent operations) plus the HEC reads below, which answer splunk_hec_tenant_required on another tenant. An operation a HEC tenant does not have answers not_available_for_splunk_hec. Every endpoint documents itself with describe=true and is listed in the REST API Reference page.

Endpoint	Purpose
hec_overview	The summary of the latest window, the pools, the endpoints needing attention, the discovered endpoints and the busiest tokens
hec_endpoints	A page of endpoints (up to 200), filters host, pool and status (up, stalled, late, silent), sorts by host, events, failures, error rate, busy, queue or refused
hec_endpoint	One endpoint, with its pipeline, output and receivers, and its splunkd API facts
hec_tokens	A page of token names
hec_token	One token by token_name: its latest window, its last 24 hours, its endpoints, its errors, the indexes refused and the client addresses
hec_lookups, hec_lookup_preview, hec_token_selection_preview, hec_token_selection_status	The Token silence editor: the lookups of a deployment, one lookup’s preview, and Test selection (start, then poll)

Example:

| trackme url="/services/trackme/v2/cribl/tenant/workspace" mode="get" body="{\"tenant_id\":\"example-hec\"}"