How TrackMe works¶
This page traces the path from raw Splunk activity to an alert in your inbox, or to a finding in your work queue. It is the “why does it behave like that?” companion to the vocabulary in Core concepts and vocabulary.
TrackMe watches in two ways. Entity tenants (DSM, DHM, MHM, VOL, FLX, FQM, WLK) give every entity a state on a monitoring cycle, described first. Dedicated tenants (User Activity Monitoring, Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream, Cribl Edge) run the same idea as a findings cycle, described in The findings cycle — dedicated tenants.
The monitoring cycle¶
TrackMe is not a live dashboard that queries Splunk when you open it. It is a set of trackers (scheduled searches) that run on a cycle and persist their results. Opening the UI just reads the last computed state. This is what lets TrackMe scale to thousands of entities and alert without anyone watching a screen.
Each cycle, for every entity tenant, the following happens:
1. DISCOVER trackers scan the tenant's indexes and find entities
(new (index,sourcetype) pairs, hosts, searches, …)
│
2. MEASURE each entity's metrics are refreshed
(delay, latency, volume, parsing quality, execution health)
│
3. DECIDE the decision maker computes, per entity:
• impact score = Σ (delay, latency, outliers, thresholds…)
• object state = green / orange / red (from the score)
│
4. CLASSIFY policies assign priority, tags, SLA; labels are evaluated
│
5. PROTECT logical groups, disruption grace, maintenance, ack
may demote an entity to blue (suppress alerting)
│
6. ALERT stateful alerting opens / updates / closes alert records
and delivers email, notable events, ingest, commands
Steps 3–6 are the heart of the product. The rest of this guide is, essentially, a detailed tour of each step.
Step 3 — how state is decided¶
The decision maker computes a single total_score by summing every active problem:
total_score = base_score (from the metrics index)
+ impact_score_delay (if the delay threshold is breached)
+ impact_score_latency (if the latency threshold is breached)
+ impact_score_outliers (from ML outlier detection)
+ threshold_scores (sum of breached dynamic thresholds)
+ …
The score is then mapped to a colour: 0 is green, < 100 is orange, ≥ 100 is
red. Because every contribution is just a number with a configurable weight, you control
exactly what makes an entity “critical” — see Part 4 for tuning impact scores and
thresholds.
Step 5 — why an entity might not alert¶
A red score does not always mean a page. Four mechanisms can hold an entity in blue (suppressed) instead:
Logical group protection — the entity’s group is mostly green, so this member is carried.
Disruption grace — the anomaly is too new; TrackMe waits to see if it persists.
Maintenance mode — the entity, tenant, or whole instance is in a maintenance window.
Acknowledgment — someone already accepted this problem.
This separation — what state (decision maker) versus whether to tell you (protection and alerting) — is deliberate, and it is why TrackMe is quiet in practice despite tracking so much.
Step 6 — how you get told¶
The stateful alerting system tracks each anomaly as it opens, persists, and closes:
It opens a stateful record when an entity first turns red (or orange, if you opt in), updates it each cycle the problem persists, and closes it when the entity goes green.
On those transitions it can deliver an email (with embedded 24-hour charts and, optionally, an AI-written incident summary), write a notable event, ingest an event, and/or run a saved-search command.
Alerting is covered in Part 6.
The findings cycle — dedicated tenants¶
A dedicated tenant has no entities and no state colour. Its scheduled jobs collect what it watches, and its policies raise findings about the objects that need a look; a healthy object has none. Each run, for every dedicated tenant:
1. COLLECT collection jobs read what the tenant watches from what Splunk
(or the Cribl Leader) already holds: inventory, activity, metrics
│
2. EVALUATE the policies, built in or your own SPL, evaluate their condition
against what was collected, object by object
│
3. RAISE each condition detected on one object is a finding, with its
evidence and a severity; a condition that clears recovers by itself
│
4. GROUP an object's findings form one Finding group: one status, one
assignee, one severity, one notification thread
│
5. ROUTE assignment rules route the group to a team; snoozes and bank
holidays hold what should wait
│
6. NOTIFY an email digest or one threaded email per Finding group, webhooks,
indexed finding events; the AI Agent can review and triage
The same separation holds as for entities: a policy decides what is wrong, and a person — or, within what the tenant allows, the AI Agent — decides what to do about it. Nothing is installed on the platforms being watched. Findings, Finding groups and everything around them are covered in Findings, Finding groups and findings actions.
Multi-tenancy and remote deployments¶
Everything above happens per tenant. A tenant can monitor the local Splunk search head, or point at a remote Splunk deployment (Enterprise or Cloud) over an authenticated, token-rotated connection with automatic failover. From the user’s point of view the experience is identical — only the data source differs. Remote deployments are covered in Part 8.
Where the AI fits¶
TrackMe’s AI features sit alongside this cycle, never in the critical path:
The AI Assistant is a context-aware chat drawer on every page, off by default until you configure a provider.
AI Advisors are on-demand agents that can inspect TrackMe’s state and propose (or, with consent, apply) tuning.
AI Routines are scheduled agents you configure to evaluate TrackMe on a cadence and then notify (email or Splunk event) or, with granted scope, act on what they find.
AI status reports are the LLM summaries optionally embedded in alert emails.
The AI Agent of each dedicated tenant type reviews findings on demand or on an hourly schedule, recommends a decision and, within what the tenant allows, applies it.
All of this is opt-in and documented in the dedicated Artificial Intelligence section.
See also
Core concepts and vocabulary — definitions for every term used above.
Findings, Finding groups and findings actions — findings, Finding groups and findings actions.
Artificial Intelligence — the AI Assistant, Advisors, Routines and the AI Agents for findings.