Cribl Stream reference

This page is the reference companion of Cribl Stream: the catalogue of built-in policies, the jobs and their schedules, the Leader request budget, the traffic ML models, the rules for silent and removed objects, the severity scale, every tenant setting, the indexed finding events, the permissions, the Configuration Guardian checks, the REST endpoints, the capacity and the troubleshooting table. The concepts shared with the other findings-based tenants (Finding groups, triage, assignment, findings actions, AI Agent) are in Findings, Finding groups and findings actions.

Policy catalogue

A Cribl Stream tenant evaluates 16 built-in policies, shown in four groups on the Policies tab. Every policy has the same common settings:

  • Enabled — on or off.

  • Severity — the severity of the findings it opens (see Severity scale).

  • Open after / Recover after — how many consecutive failing checks open a finding and how many healthy checks recover it (1 to 12). Metric policies count five-minute intervals; Worker and Leader policies count checks at least five minutes apart, so repeated manual collections never shorten a confirmation.

  • Worker Groups — the Worker Groups it applies to (empty = all).

  • Selected objects — the objects it applies to (up to 128; empty = all).

  • Bank holidays — Hold new findings or Keep opening findings while a bank holiday of the tenant’s calendars is active and the tenant suppresses new alerts on holidays. Default: Traffic outliers holds, every other policy keeps opening findings; the four Leader policies have no such setting. A finding already open keeps being evaluated.

Threshold policies are bad at or above the threshold, good at or below the recovery threshold, and hold their current condition in between. Missing data is never turned into a zero, except by the Traffic outliers policy for silent objects (see Silent and removed objects). A saved policy applies from the first collection interval after the save.

Cribl Leader

Policy

What it detects

Default

Severity

Leader availability

The configured Leader API cannot be reached or used.

On

Critical

Leader CPU

Operating-system CPU of each Leader node.

On

Medium

Leader memory

Memory in use on each Leader node.

Off

Medium

Leader disk

Used space of each filesystem a Leader node reports.

On

Medium

Data flow

Policy

What it detects

Default

Severity

Reported health

A Source or Destination Cribl reports degraded or unhealthy.

On

Medium → Critical

Traffic outliers

Traffic outside the range its traffic ML model expects.

On

Medium

Expected traffic

A selected or automatically expected object carries no traffic.

On

Medium

Pipeline errors

Pipeline errors at or above the threshold.

On

Medium

Destination backpressure

A Destination reports backpressure or is blocked.

On

High → Critical

Worker processes

Policy

What it detects

Default

Severity

Process CPU

CPU of each worker process.

On

Medium

Worker machines

Policy

What it detects

Default

Severity

Worker machine CPU

Operating-system CPU of each Worker.

On

Medium

Worker machine memory

Memory in use on each Worker.

Off

Medium

Worker machine disk

Used space of the filesystem Cribl reports for each Worker.

On

Medium

Worker availability

A Worker is disconnected or its heartbeat is too old.

On

High

Worker version drift

A connected Worker does not run the running Leader’s version.

On

Low

Worker configuration drift

A Worker does not run its Worker Group’s configuration.

On

Low

Default settings

What each policy applies to, its checks to open and to recover, and its own settings with their defaults.

Policy

Applies to

Open / recover

Settings and notes (defaults)

Leader availability

The Leader API

2 / 2

Fails on the network, the TLS certificate or host name, authentication, an unavailable service or a malformed answer. A rate limited answer is unknown, never a failure.

Leader CPU

Leader node

3 / 2

Threshold 90 %, recovery 80 %.

Leader memory

Leader node

3 / 2

Threshold 90 %, recovery 80 %. File cache included (see the note below).

Leader disk

Leader node filesystem

3 / 2

Threshold 90 %, recovery 80 %, per named filesystem.

Reported health

Source, Destination

3 / 2

Critical when Cribl reports it unhealthy, the policy severity when degraded. Only objects that carried traffic in the last 24 hours, are selected, or are already failing (Include resources without traffic: off).

Traffic outliers

Source, Destination, Pipeline, Route (per measurement)

2 / 2

Detecting models only. Direction: drops and spikes (or drops only, spikes only; also per model). The silence of an object counts as zero traffic, under guards.

Expected traffic

Source, Destination, Pipeline, Route

3 / 1

Opens nothing until objects are selected or expected. A new tenant uses Selected + automatic: objects qualify after 7 days; Sources and Destinations.

Pipeline errors

Pipeline

3 / 2

Threshold 1 error per interval.

Destination backpressure

Destination

2 / 2

Critical when fully blocked. At most one finding per Destination and condition.

Process CPU

Worker process

3 / 2

Threshold 90 %, recovery 80 %; statistic Average (or Peak).

Worker machine CPU

Worker

3 / 2

Threshold 90 %, recovery 80 %.

Worker machine memory

Worker

3 / 2

Threshold 90 %, recovery 80 %. Cache included (see the note below).

Worker machine disk

Worker filesystem

3 / 2

Threshold 90 %, recovery 80 %.

Worker availability

Worker

3 / 2

Heartbeat limit 600 s (60 to 3,600).

Worker version drift

Worker

3 / 2

Compared with the version of the running Leader.

Worker configuration drift

Worker

3 / 2

Compared with the Worker Group’s configuration version.

Note

Why memory policies are off. The memory the operating system reports as used includes its file cache, which fills a healthy Linux host. Leader memory and Worker machine memory are available but not enabled by default; enable them knowingly, with a threshold that suits your hosts.

Each policy’s editor also has an Actions tab (drilldowns, links and webhooks offered on its findings) and an Automations tab (rules that run an action automatically): see Findings actions.

Severity scale

Cribl Stream findings use the TrackMe scale Informational, Low, Medium, High, Critical.

Rule

Effect

Policy severity

Each policy has its default severity (catalogue above), which a TrackMe administrator of the tenant can change.

Raised by the condition

Reported health: a Source or Destination Cribl reports unhealthy is Critical. Destination backpressure: a fully blocked Destination is Critical. Such a raise is an escalated transition.

Operator override

Set severity on a finding or a Finding group overrides the policy’s; Use policy severity restores it.

Finding group

The highest severity of its failing members, unless overridden on the group.

Settled findings

A recovered, ended or resolved finding is shown Informational; the severity its episode reached stays recorded (peak_severity).

Earlier builds

A severity stored as warning reads Medium.

Notifications filter on the minimum severity (Medium and above by default).

Jobs and schedules

Each tenant is served by scheduled reports, each with its own lock and run receipt. Schedules are offset per tenant (from the tenant identifier), so several tenants reading the same Leader never start in the same minute.

Job

Schedule

What it does

Live metrics

every 5 minutes

Checks the Leader API, reads the newest settled five-minute interval of every metric family, scores traffic against the models, publishes the snapshot every tab reads, refreshes the Workers and the Leader resources, evaluates every policy, then delivers the queued notifications.

Inventory

every 10 minutes; a new cycle about hourly

Reads the configuration and the Workers from the Leader. A new cycle starts once the previous one is 50 minutes old; a Worker Group whose configuration version did not change is confirmed without a read (and re-read at least once a day); at most 10 Worker Groups are read per run, the changed ones first.

Traffic ML models

twice an hour

Learns the models when a pass is due (every 6 hours, or while bootstrapping); returns at once otherwise. After repeated failures it waits 1, 2, 4 then 6 hours.

Monitoring policies

on demand only

Evaluates the policies again on the newest snapshot. The scheduled evaluation is the one that ends every live run.

AI Agent

hourly, when enabled

Scheduled triage by the Cribl Stream AI Agent (off by default; the report stays unscheduled until it is enabled).

Manual collection. Settings → Collection & models → Scheduled collection is on by default; off, the reports are kept but unscheduled and Collection health reads Manual collection.

Run now. Collect now (status strip) runs the live job; Run now on each Collection health card runs that job; Read every Worker Group again (Inventory card) starts a cycle that reads every Worker Group regardless of its configuration version, for edits saved on the Leader but not yet deployed. Cooldowns per tenant, whoever asks: inventory 120 s, live 60 s.

Changing the source. Changing the connection, the metric index, the metric prefix or the Worker Groups starts a new source: collection and model learning restart, and the findings of the previous source close. Changing the schedule or the learning history keeps everything.

Leader request budget

TrackMe is gentle with the Cribl Leader at any size. Every caller that reads one Leader — every tenant, every job, the connection test, the wizard discovery — draws from one shared budget for that Leader:

  • Requests per hour — 1 200 by default (120 to 20 000), a setting of the Cribl connection.

  • Burst — 150 by default (10 to 2 000), also on the connection.

  • When several connections reach the same Leader, the lowest values apply.

  • At most two runs read one Leader at a time, at most 8 requests per second together.

A run that would exceed the budget is deferred and resumes exactly where it stopped (inventory page, Worker directory page, machine-sample position): protection, not failure. When the Leader answers slow down (HTTP 429 with a wait), every later caller waits for it; after three failing runs in a row the Leader is left alone for 60 seconds, doubling up to 30 minutes. The Collection health Cribl Leader card shows the requests of the last hour against the budget, the last run’s requests and data received, and any requested wait.

Only the live and inventory jobs, the connection test, the wizard discovery and the live status of a Source or Destination read the Leader. Every tab, Topology Studio, the policies and the model training read what TrackMe already stored.

Traffic ML models

Aspect

Behaviour

What gets a model

One model per object and measurement, all worker processes combined: Sources (events in, bytes in), Destinations (events out, bytes out), Pipelines and Routes (events in, events out). Up to 20 000 models per tenant; the busiest series get one first.

How it learns

A normal distribution per hour of the day, then per weekday and hour once 28 days of history are learned. Recent days count more (half-life 14 days by default).

Learning history

At creation, the models learn from the history already in the metric index, up to the Model learning history (30 days by default, 0 to 90), bounded by the index’s retention.

Retraining

No periodic full retrain: a pass every 6 hours reads only the intervals since the last one and merges them in. Each run has a bounded time budget and continues at the next run where it stopped.

Relearn

Relearn one model (or a selection) or Relearn models (all) re-reads the history. TrackMe administrators of the tenant.

States

Learning (less than a day, or hours not covered), Low confidence (range shown, opens no finding, until 7 days), Detecting (opens findings), No data (the object stopped reporting).

Expected range

Expected value ± z deviations: Normal 3.0, Low 4.0, High 2.5 (custom 2.0 to 6.0); deviations smaller than 10 % of the expected value or 1 event or byte per five minutes are ignored.

Per-model settings

Up to 500 models with their own settings (Tune model): training days (7 to 90), direction, pause (closes its Traffic outliers findings), up to 20 excluded periods.

Not an anomaly

Excludes the episode (from an hour before it opened to its recovery, 7 days at most) from what the model learns, relearns it at the next training run and dismisses the finding. Power and admin roles of the tenant; for non-administrators the excluded periods of a model total at most 7 days and never leave less history than the model needs to open findings.

Bank holidays

With the tenant’s Exclude from ML training effect on (the default), the hours of its bank holidays are not learned, like an excluded period, by training, the history bootstrap, a relearn and the tuning simulation. From the next training run on; no automatic relearn. With Suppress new alerts on, the policies set to hold (Traffic outliers by default, see Policy catalogue) open no new finding while a holiday is active (reason Bank holiday); open findings keep being evaluated. See Bank holidays.

Expected traffic: automatic inclusion

Rule

Value

Modes

Selected (only the objects you select) or Selected + automatic (default of a new tenant; tenants created on earlier builds keep Selected).

Qualify after

An object is expected automatically once it carried traffic with no gap longer than 10 minutes for the whole window: 7 days (default) or 2 days.

Resource types

Sources and Destinations by default; Pipelines and Routes optional.

Stops expecting

After 7 days of silence, when the object is removed from Cribl, or when it leaves the scope.

Per object

Exclude, Include again, Keep as selected (adds it to the selected objects).

Silent and removed objects

Rule

Value

Shown under No data

From the second missing five-minute interval.

Silence counted as zero traffic

By Traffic outliers only, and only when the Inventory is current and lists the object, monitoring is on, its family is collected, its Worker Group reports, it carried traffic within 24 hours, its model is Detecting (and watches drops), and no mass silence is in progress. A silence can open or keep a finding, never recover one.

Many objects stopping together

Most objects of a type in one Worker Group stopping in the same interval is a mass silence: not counted as zero, reported in Collection health (objects stopped together).

Retention of a silent object

Removed 30 days after its last measurement (7 days for a worker process), never while a policy selects it or an unresolved finding names it.

Removed from Cribl

Missing from two consecutive complete Inventory readings: its findings close at once (removed from Cribl), its silence is never counted, it leaves No data and is purged after 7 days.

Mass removal

More than half of a Worker Group’s objects of one type (at least 5) missing at once are held, not removed, and noted in Collection health.

Forget this object

Purges a silent or removed object at once. TrackMe administrators of the tenant, up to 100 objects per request, audited.

Findings lifecycle

Aspect

Value

Condition

Failing, Recovered, Awaiting data (unknown), Superseded.

Status

Open, Investigating, Resolved, Dismissed. A measured recovery resolves an open or investigating finding; a new failure reopens it.

Dismiss

Until it recovers (default), for 1, 7, 30 or 90 days (then back to open), or permanently.

Snooze

Up to 30 days; not a status. Stops notifications and colours the object blue in Topology Studio.

Automatic closures

Source or connection changed; object not observed or not evaluated for 24 hours; traffic no longer reported; Worker no longer listed; monitoring turned off; model paused; removed from Cribl.

Finding groups

The findings of one object form one group automatically (up to 50 members); merge, add, remove and ungroup by hand. See Finding groups.

Retention

Closed findings and Finding groups are kept 90 days (20 000 of each at most). At most 5 000 unresolved subjects per tenant.

Settings reference

Settings are saved by a TrackMe administrator of the tenant, except the policies (the tenant’s admin roles) and the monitoring state (the tenant’s power and admin roles).

Section

Setting

Default

Meaning

Source & Worker Groups

Cribl connection

—

Fixed once saved: monitoring another deployment is another tenant.

Cribl metric index

—

The metric index that receives Cribl’s internal metrics.

Metric prefix

cribl.logstream.

Letters, digits and dots, ending with a dot.

Worker Groups

all discovered

1 to 100.

Repair workspace

—

Recreates the tenant’s stores and reports; keeps findings and settings.

Collection & models

Scheduled collection

On

Off = manual collection.

Model learning history

30 days

0 to 90 days; applies the next time the models are relearned.

Traffic ML models

Sensitivity

Normal (3.0)

Low 4.0, High 2.5, or custom 2.0 to 6.0 deviations.

Seasonality

Automatic

Hour of day, weekday and hour, or none.

Weekday pattern after

28 days

History needed before the weekday × hour pattern is used.

Recent days count more

14 days (half-life)

How fast older days fade.

Open findings once

7 days of history

When a model moves from Low confidence to Detecting.

Ignore deviations smaller than

10 % and 1 event or byte per 5 minutes

Smallest deviation that counts.

Notifications

Enable notifications

Off

Email delivery of finding transitions.

Events

opened, reopened, added to the group, escalated, recovered, dismissal ended, closed

Other transitions can be added.

Minimum severity

Medium and above

Low and above, Medium and above, High and above, Critical only.

Policies

all

Limit the emails to some policies.

Delivery account

the first configured account

A TrackMe email delivery account, or Default (localhost MTA).

Recipients

—

Up to 50, filtered by the account’s allowed domains.

Mode

Digest

One digest per live run, or one threaded email per Finding group.

Include charts

On

The finding’s chart and key measurements of 24 hours, inline.

AI Agent

Scheduled triage

Off

Hourly review of open findings.

Mode

Inspect (recommend only)

Or Act (apply decisions).

AI provider

First available provider

The provider of the scheduled runs.

Review interval (days)

1

0 to 365; a reviewed finding is skipped this long unless a new episode starts.

Findings per run

10

1 to 50.

Time budget per run (seconds)

3 000

60 to 86 400.

Custom instructions

empty

Plain English the agent follows on every run, asked or scheduled (at most 4000 characters); Evaluate… tries an unsaved draft in inspect mode and writes nothing.

Allowed actions

Mark investigating, Dismiss for a time, Not an anomaly

Resolve and Stop monitoring are off by default.

Longest dismissal (days)

7

1 to 90.

Proven inactivity (days)

7

1 to 365; how long an object must be inactive before the agent may stop monitoring it.

Assignment

Assignment groups, rules

none

See Assignment and automatic routing.

Finding actions

Actions, automation rules

none

See Findings actions.

Policies tab

Every policy, Expected traffic mode

see Policy catalogue

Tenant admin roles.

Inventory tab

Exclusion rules, objects turned off

none

Up to 200 rules and 20 000 objects turned off; tenant power and admin roles.

Cribl connection

Verify SSL certificate, CA certificate chain, requests per hour, burst

On, —, 1 200, 150

Configuration → Cribl connections (Cribl connections).

The Source & Worker Groups section also has two settings available through the REST API only: the bucket span (300 s) and the settle delay (120 s).

Finding events

Every transition is written as one JSON event in the tenant’s notable index, whatever the notification settings:

  • sourcetype=trackme:cribl:finding — one event per finding transition (event_kind=finding_transition);

  • sourcetype=trackme:cribl:finding_group — one event per Finding group transition (event_kind=finding_group_transition).

Main fields: tenant_id and object_category (cribl_stream), finding_id, group_id, transition and transition_label (Finding opened, Severity raised, Measurement recovered, Closed automatically, …), transition_id (deduplicate on it), policy_id and policy_name, severity, display_severity, peak_severity, condition, status, reason, family, worker_group, entity_name (the object), kpi, value, expected_lower, expected_upper, assignee, actor.

index=trackme_notable sourcetype=trackme:cribl:finding tenant_id="<tenant>"
    transition IN ("opened", "reopened", "escalated") severity="critical"
index=trackme_notable sourcetype=trackme:cribl:finding_group tenant_id="<tenant>"
| stats latest(_time) as last_at latest(status) as status latest(severity) as severity
        latest(object) as object latest(worker_group) as worker_group by group_id
| where status IN ("open", "investigating")

Use them for Splunk alerts, Enterprise Security correlation searches or SOAR playbooks.

Permissions

Permissions combine a TrackMe capability with the tenant’s roles. Members of admin or sc_admin can do everything on every tenant. A TrackMe administrator of one tenant has no right on another.

Tier

Who

Actions

Read

trackmeuseroperations and any role of the tenant

Every tab, every chart, the investigation drawer, the AI Assistant, drilldown and link actions.

Triage

trackmepoweroperations and the tenant’s power or admin role

Status, dismiss, snooze, severity, assignment, notes, Finding groups, Not an anomaly, monitoring on / off and exclusion rules, Stop monitoring this object, Ask the AI Agent and apply its recommendations, send a webhook action.

Policies

trackmeadminoperations and the tenant’s admin role

Edit policies, the Expected traffic mode and its automatic list.

Manage (a TrackMe administrator of this tenant)

trackmeadminoperations and the tenant’s admin role

Source settings, Collect now and Run now, model tuning and relearn, Repair workspace, Forget this object, notifications, AI Agent settings, assignment and finding actions.

Tenant owner. The tenant’s reports run as its owner: nobody, a Splunk administrator, or an account holding trackmeuseroperations, trackmepoweroperations, trackmeadminoperations, schedule_search, list_settings, list_storage_passwords and admin_all_objects. The scheduled AI Agent needs an owner that is a Splunk administrator account (not nobody).

Cribl connections are managed in Configuration → Cribl connections (Cribl connections).

Configuration Guardian checks

cribl_collection_health (tenant scope, warning; evaluated at every tenant health cycle) is raised with one or more of these problems. A disabled tenant, a tenant without a configured source, or a tenant on manual collection reports nothing.

Problem code

Meaning

live_never_ran / live_not_running

No live metrics run since the source was configured, or none finished in the last 20 minutes.

live_failing

Live runs happen but publish nothing, or their traffic scoring fails, for 30 minutes.

metrics_not_arriving

No Cribl metric reached the metric index for 30 minutes.

worker_groups_without_metrics

Selected Worker Groups send no metric.

inventory_never_ran / inventory_not_running

No inventory run since the source was configured, or none finished for 3 hours.

inventory_failing

Inventory runs do not complete for 90 minutes.

inventory_not_published

The inventory was last published more than a day ago (runs keep waiting for the Leader or do not finish).

inventory_cycle_slow

A full inventory cycle takes more than a day under the Leader request budget.

worker_sampling_slow

Worker machine samples do not keep up under the Leader request budget.

worker_directory_stale

The Worker directory is not refreshed (30 minutes): Worker availability and versions are out of date.

presence_failing

The directory of objects that stopped reporting is not kept up to date.

objects_stopped_together

Many objects stopped reporting in the same interval (for 30 minutes).

policies_failing

Policy evaluation does not complete for 30 minutes.

models_failing

Model training does not complete for 90 minutes.

capacity_reached

A capacity limit stopped collection, scoring or training, or some traffic series have no model (more series than the tenant keeps models for).

notification_delivery_failing

Finding emails fail for 30 minutes.

Notes shown in Collection health that never raise the check: candidates capped, objects removed together, Expected traffic capped or out of time.

System checks on Cribl connections:

  • cribl_connection_health — the daily connection probe fails: warning, critical after 24 hours; visible to administrators.

  • cribl_connection_tls_unverified — a self-managed connection has certificate verification turned off: warning.

Also relevant: insufficient_tenant_owner_capabilities (the tenant owner lacks a required capability), ai_automation_owner_is_system_user (scheduled AI Agent with a nobody owner), findings_action_delivery_failing (webhooks or other findings actions failing) and topology_view_stale_references. See Configuration Guardian.

REST endpoints

All endpoints are under /services/trackme/v2/ and described in the REST API Reference (describe=true). Reads run with the caller’s session; changes are authorized on the caller’s session first, then applied and audited.

Group

Operations (cribl/tenant/<operation>)

Workspace and reads

workspace, evidence, inventory, workers, summary, health_summary, findings, finding, finding_groups, finding_group, traffic_models, traffic_model, tuning, presence, policies, expected_auto, monitoring, object_status, agent_settings, agent_findings, agent_evidence, agent_decisions

Triage

triage, group_triage, group_members, not_anomaly, monitoring (POST), monitoring_rules, monitoring_preview, finding_stop_monitoring, agent_decide, agent_review

Policies

policies (POST), policy_preview, expected_auto_preview, expected_auto_override

Manage

configure, collect, collect_result, models, repair, tuning (POST), forget_objects, notifications (GET and POST), notification_test, agent_settings (POST)

Other

cribl/admin/connection_health (administrators); the AI Agent jobs ai_findings_agent/agent, agent_status, agent_cancel; findings actions findings_actions/*; the Cribl connections through the configuration endpoint trackme_cribl_connection.

Capacity

Limit

Value

Monitored objects per tenant (Sources, Destinations, Pipelines, Routes, worker processes in the snapshot)

About 20 000; split a larger deployment across tenants by Worker Group.

Worker Groups per tenant

100

Traffic ML models

20 000

Models with their own settings

500

Unresolved findings (subjects)

5 000; beyond, new failures are refused while every recorded one keeps being evaluated.

Closed findings and Finding groups kept

20 000 each, 90 days

Selected objects per policy

128

Exclusion rules / objects turned off one by one

200 / 20 000

Notification recipients

50

Forget this object

100 objects per request

Models need about a week of history before they open findings, unless the metric index already holds that history at creation.

Logs

Every line starts with tenant_id="<tenant>".

Log

Sourcetype

Contains

trackme_cribl_collect.log

trackme:custom_commands:trackmecriblcollect

One summary line per run; WARNING when a run did not complete; Cribl collector failure for an internal error.

trackme_cribl_tune.log

trackme:custom_commands:trackmecribltune

Failures of the tuning simulation.

trackme_cribl_notify.log

trackme:custom_commands:trackmecriblnotify

One line per test email.

trackme_rest_api_cribl.log

trackme:rest_api

Degraded paths of the REST API.

index=_internal sourcetype=trackme:custom_commands:trackmecriblcollect tenant_id="<tenant>"
index=_internal sourcetype=trackme:custom_commands:trackmecriblcollect tenant_id="<tenant>" job="live"
| stats count, latest(_time) as last, values(reason) as reasons by complete

Tokens, passwords, email addresses, request bodies and search strings are never logged.

Troubleshooting

Symptom

Likely cause

Fix

The wizard proposes no metric index

Cribl’s internal metrics do not reach a Splunk metric index, or they use another prefix.

Route Cribl’s internal metrics to a Splunk metric index; set the metric prefix your deployment uses.

Metrics not arriving in Collection health

The metrics stopped reaching the index, or the index or prefix is wrong.

Check the Cribl Splunk Destination and the metric index; compare the prefix in Settings.

Worker Groups without metrics

A selected Worker Group does not send its internal metrics.

Enable the internal metrics on that Worker Group, or deselect it.

Leader · unavailable, a Critical Leader availability finding

Network, certificate, credentials or service problem on the Leader API (the tooltip names it).

Certificate not trusted or host name mismatch: add the CA chain to the connection. Credentials refused: update them. Then test the connection.

Leader · rate limited, deferred runs

The Leader asked TrackMe to slow down, or the request budget is spent.

Wait; lower the connection’s requests per hour if the Leader is shared; check the Cribl Leader card in Collection health.

Inventory cycle slow / Worker sampling slow

The deployment needs more requests than the Leader budget allows.

Raise the connection’s requests per hour if the Leader can take it, or split the deployment across tenants.

Traffic outliers open nothing

Models are still Learning or Low confidence.

Wait until they are Detecting (7 days of history), or recreate with a longer Model learning history.

Too many Traffic outliers findings

The models are too sensitive for this traffic.

Use Not an anomaly on expected episodes; lower the sensitivity or raise Ignore deviations smaller than; limit the direction to drops.

Worker memory findings on healthy hosts

Memory used includes the file cache.

Keep Worker machine memory off, or raise its threshold.

A deleted object still shows under No data

The Inventory has not seen it missing twice yet, or it was turned off.

Wait for the next inventory cycle, or Forget this object.

The scheduled AI Agent does not run

AI Assistant disabled, no provider, or a nobody tenant owner.

Enable the AI Assistant, configure a provider, and give the tenant a Splunk administrator owner.

No email

Notifications off, events or severity filtered out, or delivery failing.

Check Settings → Notifications, then Send test email; see notification_delivery_failing.

See also