Cribl Stream & Cribl Edge

Cribl sits in the middle of the data path: when a Cribl Leader, a Worker, an Edge Node, a Source or a Destination has a problem, the data behind it slows down, backs up or stops. TrackMe monitors Cribl in complementary ways. Pick the one that matches what you want to watch:

What you want to watch	What to use
A Cribl Stream deployment — its Leader, Worker Groups, Workers, Sources, Destinations, Pipelines and Routes	The Cribl Stream tenant: Cribl Stream
A Cribl Edge fleet — its Fleets, Edge Nodes, Sources and Destinations	The Cribl Edge tenant: Cribl Edge
The data sources and hosts whose logs pass through Cribl	DSM and DHM with Cribl signals: Monitor data sources and hosts with Cribl signals
Custom Cribl monitoring on your own SPL	Flex Objects and the Cribl API add-on: Cribl API add-on and Flex use cases (annex)

The Cribl Stream and Cribl Edge tenants are generally available in TrackMe 2.4.19. Both start from a Cribl connection (Cribl connections), declared once in TrackMe and shared by every tenant that reads the same Cribl deployment.

How the Cribl tenants work

Dedicated tenant types. A Cribl Stream tenant and a Cribl Edge tenant are dedicated tenant types, not entity components: they have no entities, no decision maker and no health score, and none of the DSM, DHM, MHM, FLX, FQM, WLK or VOL components. Each has its own workspace (Tenant Home), its own collection jobs, a catalogue of built-in policies and the findings those policies raise.

Read-only, nothing installed on Cribl. TrackMe reads what already exists and never changes anything in Cribl:

• the Cribl Leader API, for the configuration (Worker Groups or Fleets, Sources, Destinations and, on Stream, Pipelines, Routes and Packs), the Workers or Edge Nodes and the deployment state. Every call is a read; the only requests sent with POST are the sign-in and, on Edge, the query of an Edge Node’s own system metrics, which changes nothing;

• for Cribl Stream, the Cribl internal metrics that Cribl already sends to a Splunk metric index. They are computed in place with mstats: nothing is copied into a TrackMe index, and the metrics cause no additional licence ingestion.

No agent, Pack or configuration is installed on Cribl, and no tracker has to be written.

Gentle with the Leader, at any size. Every TrackMe caller that reads one Leader — every tenant, every job, the connection tests — draws from one shared Leader request budget (1 200 requests per hour by default, a setting of the connection). A run that would exceed it waits for the next one and resumes where it stopped instead of bursting; unchanged Worker Groups are confirmed without a read; Cribl Edge reads the node list in bulk, never one call per node. When the budget cannot keep up, Collection health says so and names the remedy.

Findings, the shared model. Both tenants raise findings — one per object, policy and measurement — with Finding groups, the TrackMe severity scale (Informational, Low, Medium, High, Critical), triage, email notifications, assignment and findings actions. The model is the same as in User Activity Monitoring and is described once in Findings, Finding groups and findings actions.

Cribl Stream and Cribl Edge at a glance

	Cribl Stream tenant	Cribl Edge tenant
Monitors	The Leader, Worker Groups, Workers and Worker processes, Sources, Destinations, Pipelines and Routes	Fleets (and Subfleets), Edge Nodes, the Sources and Destinations of each Fleet, the Leader’s availability
Reads	The Cribl internal metrics in a Splunk metric index (required) and the Leader API	The Leader API; optionally a System Metrics index for the CPU and memory of the Edge Nodes
Built-in policies	16	11
Traffic ML models, Expected traffic	Yes	No
AI	AI Assistant, Concierge actions and the Cribl Stream AI Agent	AI Assistant and the Cribl Edge AI Agent
Collection jobs	Inventory, live metrics, traffic model training	Inventory, live
Size of one tenant	Up to about 20 000 monitored objects	Any fleet size
Shared by both	Cribl connections, the Leader request budget, findings and Finding groups, notifications, assignment, findings actions, Collection health, Topology Studio

Cribl connections

A Cribl connection tells TrackMe how to reach one Cribl Leader: a Cribl.Cloud workspace or a self-managed Leader. It is the prerequisite of every Cribl Stream and Cribl Edge tenant. Connections live in Configuration → Manage Cribl connections; the credential is stored encrypted in Splunk and never shown again, and in a Search Head Cluster the connections replicate to every member.

Adding a connection

Click Add connection, give the connection a name (lowercase letters, digits and underscores, starting with a letter; it cannot change later) and choose the deployment:

	Cribl.Cloud	Self-managed Cribl
Where	Cloud organization ID (the ID, not the display name) and Cloud workspace (main for the default workspace)	Leader URL: an HTTPS origin only, for example https://cribl.example.com:9000 (no path, credentials or query)
Credential	An API credential: Client ID and Client secret (OAuth client credentials), created in Cribl.Cloud (below)	A Cribl Username and Password: a service account that signs in with a user name and password, a Cribl local user when the Leader uses SSO (below)
How TrackMe signs in	Exchanges the client ID and secret at login.cribl.cloud for a bearer token, valid 24 hours	Signs in at /api/v1/auth/login on the Leader for a bearer token, valid for the Leader’s Auth token TTL (1 hour by default)
TrackMe reaches	login.cribl.cloud for the sign-in, then the workspace URL https://<workspace>-<organization>.cribl.cloud	The Leader URL
Leader certificate	Always verified against the public certificate authorities	Verified by default; a custom CA chain can be added (below)

The Splunk search heads that run the tenants’ collection must reach these addresses over HTTPS. HTTP proxies and client certificates (mutual TLS) are not supported.

In both modes every API request carries a bearer token, never the credential itself. TrackMe obtains the token, keeps it while it is valid and signs in again on its own when it expires: nothing needs renewing by hand.

Cribl.Cloud: the API credential

1. In Cribl.Cloud, as an organization Owner or Admin, open Organization → API Credentials and select Add Credential. Name it after its use, for example trackme.

2. Under Organization Permissions, choose the User role, then under Workspace Access give it the workspace and the product the tenants monitor (Cribl Stream, Cribl Edge or both), with the most restricted permission that can read what TrackMe needs (What the Cribl account needs). The Admin and Owner roles give administrator access to every workspace of the organization: avoid them. On Cribl.Cloud plans without the Organization Permissions selector, every API credential has the Admin role.

3. Copy the Client ID and the Client secret into the TrackMe connection. The Cloud organization ID and the Cloud workspace are the two parts of your workspace URL, https://<workspace>-<organization>.cribl.cloud.

The client secret never leaves Splunk except for the sign-in at login.cribl.cloud. When the credential is rotated or disabled in Cribl.Cloud, update the connection.

Self-managed Cribl: the user account

A self-managed (customer-managed) Cribl deployment has no API credential and no API token to create: Cribl reserves the client ID and secret to Cribl.Cloud. Its API accepts one way in, a sign-in with a user name and password at /api/v1/auth/login, which returns a short-lived bearer token. This is not HTTP Basic authentication:

• TrackMe sends the user name and password only to that sign-in endpoint, over HTTPS (with the Leader certificate checks described below). Every other request carries the bearer token.

• The token lives for the Leader’s Auth token TTL (Settings → Global → General Settings → API Server Settings → Advanced, 1 hour by default). TrackMe signs in again when it expires or when the Leader refuses it. There is no need to raise the TTL for TrackMe.

• The password is stored encrypted in Splunk and is never shown again.

The account TrackMe uses:

• Signs in with a user name and password. With Local, Splunk or LDAP authentication on the Leader, that is an account that can sign in on the Leader’s login page with a user name and password.

• Is a Cribl local user when the Leader uses SSO. Single sign-on accounts (OpenID Connect or SAML) cannot sign in through the API. Create a local user for TrackMe and make sure Allow login as Local User is on (Settings → Global → Access Management → Authentication); otherwise the connection test fails on the sign-in.

• Is a dedicated service account, for example trackme, rather than a person’s account, so that a password change or a departure does not stop the collection.

• Reads, never writes: give it the most restricted role that can read what TrackMe needs (What the Cribl account needs).

When the password changes, or expires under your Cribl password policy, update the connection: until then its tests fail and the Configuration Guardian reports cribl_connection_health.

Leader certificate (TLS)

HTTPS is mandatory in every mode. For a self-managed Leader:

• Verify SSL certificate (on by default) validates the Leader certificate: its CA chain and its host name. Keep it on.

• CA certificate chain (optional): when the Leader certificate is signed by a private CA, or is self-signed, paste or upload (Upload PEM file) the root CA certificate and any intermediate CA certificates — or the self-signed certificate itself. The chain is added to the public certificate authorities, so publicly trusted Leaders keep working. Certificates only: a private key is refused (64 KB and 20 certificates at most). The Leader URL host name must match the certificate.

• Turning Verify SSL certificate off keeps the traffic encrypted but no longer checks the Leader’s identity. Use it only as a last resort, when the certificate cannot be validated even with its CA chain. It is refused for Cribl.Cloud.

The table shows the posture of each connection: Certificate verified, Certificate verified · custom CA or Certificate not verified.

Warning

While a connection has Verify SSL certificate off, the Configuration Guardian raises a cribl_connection_tls_unverified warning for it: a host impersonating the Leader would receive the connection’s credentials. Add the Leader’s CA chain and turn verification back on to clear it.

Leader request budget

Setting	Default	Range	Meaning
Requests per hour	1 200	120 – 20 000	How many requests per hour TrackMe may send to this Leader, shared by every tenant and job that reads it.
Burst	150	10 – 2 000	The most requests TrackMe may send at once after a quiet period.

When several connections reach the same Leader (a Cribl Stream and a Cribl Edge tenant on the same Cribl.Cloud workspace, for example), they share one budget and the lowest values apply. Raise the budget when a tenant’s Collection health reports that the inventory or the Worker sampling cannot keep up.

Testing connections

• Save and test stores the connection, then signs in to Cribl and reads the Worker Groups and the Workers. Save alone stores it without a test.

• Test connectivity (row menu) tests one saved connection; Test all tests every enabled connection one after the other (the first 20 when there are more).

• The Connectivity column shows the last result (Connected, Partial discovery, Test failed, Not tested, or Test again when the connection changed since). View result explains a failure and what to do next.

• A connection can be disabled: it is kept, but cannot be tested or used.

Who can manage connections

Managing connections (create, edit, test, delete) is a Splunk administrator task: the connection configuration and its encrypted credential are restricted to the admin and sc_admin roles, and TrackMe also checks its own administrator capability (trackmeadminoperations), which those roles hold.

Nobody else needs access to the connections. A tenant’s users never read the credential: when they act in the workspace, TrackMe checks their tenant role first, then uses the connection on their behalf.

What the Cribl account needs

Read access to the Cribl API of the product the tenant monitors:

• the Worker Groups or Fleets and the Workers or Edge Nodes (/api/v1/products/stream/… or /api/v1/products/edge/…);

• the configuration of each Worker Group or Fleet (Sources and Destinations, plus Pipelines, Routes and Packs on Stream);

• the Leader’s system information;

• the Leader-proxied reads of a Worker or Edge Node: its system information and the status of its Sources and Destinations, and on Edge its own system metrics.

TrackMe never asks for write access and never changes anything in Cribl. TrackMe does not prescribe a specific Cribl role: give the account the most restricted role of your Cribl deployment that can read these. A forbidden test result means the account cannot read part of them.

Licence

Cribl connections and the Cribl Stream and Cribl Edge tenants need a valid Enterprise, Unlimited, trial or Developer licence. They are not available with the Foundation edition or the Foundation trial. Without a valid licence the connections can still be listed, but not created, changed or tested.

Configuration Guardian

Two system checks watch the connections. They are shown to administrators only; the connections page links to them (Cribl connection alerts in Configuration Guardian):

Check	What it reports
cribl_connection_health	A connection whose daily connectivity test fails, is incomplete or has no recent result. A warning at first, critical when the failure persists. Cleared by the next complete test.
cribl_connection_tls_unverified	An enabled self-managed connection with Verify SSL certificate off. One warning per connection, cleared when verification is back on.

Each tenant has its own collection check as well (cribl_collection_health), described in its pages. See Configuration Guardian.

Choosing your tenants

• One connection per tenant. A tenant reads one Cribl connection, chosen at creation; changing it later starts a new source. A connection can serve several tenants.

• Stream or Edge, never both. A tenant is a Cribl Stream tenant or a Cribl Edge tenant. A Cribl.Cloud workspace that runs both products takes one tenant of each, on the same connection.

• Several deployments, several tenants. One tenant per Cribl Stream deployment or Cribl Edge Leader you monitor.

• Size. A Cribl Stream tenant monitors up to about 20 000 objects (Sources, Destinations, Pipelines, Routes and Worker processes): above that, split the deployment across tenants by Worker Group. A Cribl Edge tenant follows a fleet of any size.

Tenants of the same Leader share its request budget: Collection health tells you when it needs raising.

Permissions

Daily operation of a Cribl tenant needs no Splunk administrator role. TrackMe decides on its own capabilities and the tenant’s roles, as for every tenant (Roles & access control):

Who	What they can do in a Cribl tenant
Tenant user role (trackmeuseroperations)	Read every tab of the workspace; ask for the live status of an object.
Tenant power role (trackmepoweroperations)	Triage findings and Finding groups (investigate, assign, snooze, dismiss, resolve, change the severity, group by hand), write notes, turn monitoring of an object on or off, Stop monitoring this object; ask the AI Agent and let it act; on Cribl Stream, Not an anomaly.
Tenant admin role (trackmeadminoperations), “a TrackMe administrator of this tenant”	Everything above, plus policies, the source settings, Collect now and Run now, notifications, assignment and findings actions settings, the AI Agent settings, Forget this object, and on Cribl Stream the traffic ML models and Expected traffic.
Splunk administrator (admin, sc_admin)	Every tenant, and the Cribl connections.

Each tier needs the TrackMe capability and one of the tenant’s matching roles: a TrackMe administrator of one tenant has no right on another. The tenant owner (the account its reports run as) is nobody or an account that holds the capabilities the creation wizard lists.

Where to go next

Cribl Stream

Create a Cribl Stream tenant and work in its workspace: findings, monitoring, traffic ML models, infrastructure, deployment, policies.

Cribl Stream

Cribl Stream reference

Collection jobs, policy catalogue and defaults, settings, capacity and troubleshooting.

Cribl Stream reference

Cribl Edge

Create a Cribl Edge tenant and follow your Fleets and Edge Nodes.

Cribl Edge

Cribl Edge reference

The Edge policy catalogue, collection, settings and troubleshooting.

Cribl Edge reference

Findings

Findings, Finding groups, triage, notifications, assignment and findings actions.

Findings, Finding groups and findings actions

Cribl signals for DSM and DHM

Monitor the data sources and hosts whose logs pass through Cribl.

Monitor data sources and hosts with Cribl signals

Flex Objects and the Cribl API add-on

Custom Cribl monitoring built on your own SPL.

Cribl API add-on and Flex use cases (annex)

Hands-on: Cribl Stream

A guided tour of a Cribl Stream tenant, step by step.

Cribl Stream

Hands-on: Cribl Edge

A guided tour of a Cribl Edge tenant, from the Fleets to one node up close.

Cribl Edge