How TrackMe works¶

This page traces the path from raw Splunk activity to an alert in your inbox. It is the “why does it behave like that?” companion to the vocabulary in Core concepts and vocabulary.

The monitoring cycle¶

TrackMe is not a live dashboard that queries Splunk when you open it. It is a set of trackers (scheduled searches) that run on a cycle and persist their results. Opening the UI just reads the last computed state. This is what lets TrackMe scale to thousands of entities and alert without anyone watching a screen.

Each cycle, for every tenant, the following happens:

1. DISCOVER     trackers scan the tenant's indexes and find entities
                (new (index,sourcetype) pairs, hosts, searches, …)
                     │
2. MEASURE      each entity's metrics are refreshed
                (delay, latency, volume, parsing quality, execution health)
                     │
3. DECIDE       the decision maker computes, per entity:
                  • impact score  = Σ (delay, latency, outliers, thresholds…)
                  • object state  = green / orange / red   (from the score)
                     │
4. CLASSIFY     policies assign priority, tags, SLA; labels are evaluated
                     │
5. PROTECT      logical groups, disruption grace, maintenance, ack
                may demote an entity to blue (suppress alerting)
                     │
6. ALERT        stateful alerting opens / updates / closes alert records
                and delivers email, notable events, ingest, commands

Steps 3–6 are the heart of the product. The rest of this guide is, essentially, a detailed tour of each step.

Step 3 — how state is decided¶

The decision maker computes a single total_score by summing every active problem:

total_score =  base_score              (from the metrics index)
            +  impact_score_delay       (if the delay threshold is breached)
            +  impact_score_latency     (if the latency threshold is breached)
            +  impact_score_outliers    (from ML outlier detection)
            +  threshold_scores         (sum of breached dynamic thresholds)
            +  …

The score is then mapped to a colour: 0 is green, < 100 is orange, ≥ 100 is red. Because every contribution is just a number with a configurable weight, you control exactly what makes an entity “critical” — see Part 4 for tuning impact scores and thresholds.

Step 5 — why an entity might not alert¶

A red score does not always mean a page. Four mechanisms can hold an entity in blue (suppressed) instead:

Logical group protection — the entity’s group is mostly green, so this member is carried.
Disruption grace — the anomaly is too new; TrackMe waits to see if it persists.
Maintenance mode — the entity, tenant, or whole instance is in a maintenance window.
Acknowledgment — someone already accepted this problem.

This separation — what state (decision maker) versus whether to tell you (protection and alerting) — is deliberate, and it is why TrackMe is quiet in practice despite tracking so much.

Step 6 — how you get told¶

The stateful alerting system tracks each anomaly as it opens, persists, and closes:

It opens a stateful record when an entity first turns red (or orange, if you opt in), updates it each cycle the problem persists, and closes it when the entity goes green.
On those transitions it can deliver an email (with embedded 24-hour charts and, optionally, an AI-written incident summary), write a notable event, ingest an event, and/or run a saved-search command.

Alerting is covered in Part 6.

Multi-tenancy and remote deployments¶

Everything above happens per tenant. A tenant can monitor the local Splunk search head, or point at a remote Splunk deployment (Enterprise or Cloud) over an authenticated, token-rotated connection with automatic failover. From the user’s point of view the experience is identical — only the data source differs. Remote deployments are covered in Part 8.

Where the AI fits¶

TrackMe’s AI features sit alongside this cycle, never in the critical path:

The AI Assistant is a context-aware chat drawer on every page, off by default until you configure a provider.
AI Advisors are on-demand agents that can inspect TrackMe’s state and propose (or, with consent, apply) tuning.
AI Routines are scheduled agents you configure to evaluate TrackMe on a cadence and then notify (email or Splunk event) or, with granted scope, act on what they find.
AI status reports are the LLM summaries optionally embedded in alert emails.

All of this is opt-in and documented in the dedicated Artificial Intelligence section.