Product introduction¶
What TrackMe is and what it watches, from your data to the licensed volume, the people, the forwarders, receivers and collection points, the HTTP Event Collector endpoints and the Cribl pipelines around it; the two ways of watching — entities and findings — how it decides and how it tells you: the vocabulary and the mental model before the live product.
about 25 minutes · 30 slides. Each slide is followed by a description of what it shows — read top to bottom, this page is the walkthrough.
Overview¶
This introduction sets the vocabulary and the mental model of TrackMe 2.4.24. Everything here maps one to one to the Product Guide on docs.trackme-solutions.com. TrackMe started with the data — is it arriving, is it good — and today it covers the platform around it too: the licensed volume of every index, the people and service accounts using Splunk, the forwarders and receivers carrying the data, the collection points running add-on and syslog inputs, the HTTP Event Collector endpoints receiving it, and the Cribl pipelines in front of it.
What breaks silently¶
Most teams come to TrackMe because of one specific pain: a critical feed died hours ago and nobody noticed; a host went silent while the index still looked busy; a field extraction drifted and a detection stopped firing; scheduled searches started skipping. And the platform around the data fails just as quietly: an index doubling its licensed volume, an account running all-time searches on a schedule nobody owns, two hundred forwarders dropping at once behind a receiver tier, an intermediate forwarder whose output pauses and holds every sender behind it, a database input that stopped collecting, a HEC token gone silent behind a load balancer that still answers, a Cribl Destination in backpressure in front of Splunk. One root cause: Splunk does not monitor itself, and you cannot watch or classify all of it by hand.
What TrackMe is¶
TrackMe discovers, maintains and monitors the availability and quality of your data. It is a control plane on top of Splunk that runs entirely inside Splunk — on-premises, Splunk Cloud or hybrid — with no outbound SaaS. It answers the questions classic Splunk monitoring leaves open, from the data itself to the platform around it: is the data flowing, is it good, is the licensed volume normal, are the searches healthy, who is doing what, is my fleet forwarding, is every input collecting, is the pipeline in front healthy, from the HEC endpoints to Cribl Stream and Cribl Edge. Who it is for: Splunk administrators proving the platform is healthy, SOC and detection engineers who depend on complete, well-parsed data, platform owners who answer for the licence and the usage, service owners reporting SLA attainment, and managed service providers.
The big picture¶
On the left, what TrackMe reads: your Splunk data, the Splunk platform itself, and Cribl. In the middle, two ways of watching. Entity monitoring is the historical core: seven components turn data feeds, hosts, metrics, the licensed volume of each index, field quality, scheduled searches and any SPL into entities with a state and an impact score. Findings monitoring is the second model, used by seven dedicated components — User Activity Monitoring, Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream and Cribl Edge: policies raise findings, grouped per object into Finding groups that you triage. Both live in Virtual Tenants. On the right, what you get from both: alerting and notifications, live maps in Topology Studio, the AI layer, and the REST API.
The vocabulary¶
TrackMe has a small, consistent vocabulary that appears on every page of the guide and in the UI. Entity monitoring: an entity belongs to a component, lives in a virtual tenant, gets its state from the decision maker; policies classify it and alerting turns critical states into notifications. Findings monitoring adds two words: a finding, raised by a policy about one object — an account, a HEC endpoint, a receiver pool, a forwarding agent, a collector’s input, a Cribl Destination — and the Finding group that gathers an object’s findings into one decision. There is no global list of entities: everything lives inside exactly one tenant.
What TrackMe monitors¶
TrackMe monitors two families, grouped by the question each answers. Seven entity components: DSM, DHM and MHM answer whether the data is arriving, VOL whether the licensed volume of each index is normal, FQM whether the data is well-formed, WLK whether the scheduled searches are healthy, and FLX monitors anything else from the SPL you write.
Beside them, seven dedicated components raise findings and Finding groups instead of tracking entities: User Activity Monitoring answers who is doing what on Splunk, Splunk HEC whether every HTTP Event Collector endpoint, pool and token is receiving, Splunk S2S whether the receivers and the intermediate forwarders accept and keep up, pool by pool, Splunk UF Fleet whether every forwarding agent is connected and its Deployment Servers are healthy, Splunk Forwarder Collectors whether every input keeps collecting, Cribl Stream whether the Cribl pipeline in front of Splunk is healthy, and Cribl Edge whether every Edge Node is where it should be. Almost every deployment starts with DSM.
One entity model¶
The shared entity model is what makes TrackMe consistent: the same object, alias, state and score shape, the same lifecycle. Classification, protection, stateful alerting, scoring and ML outliers are documented once and apply to all seven components. Where they differ: variable delay and lagging classes are DSM and DHM; data sampling and elastic sources are DSM; dynamic thresholds are FLX and FQM; versioning and orphan detection are WLK; VOL has one entity per index, fed incrementally from the license usage log, with ML outliers as its state.
Entity monitoring in practice¶
Tenant Home lists every data source the hybrid tracker discovered, already scored: the counts by state and priority on top, then the entities with their status, impact score, priority, delay and latency against their thresholds. Here several feeds are orange with a score of 48.
The status message of one of them explains why in plain words: its ingestion latency breached the threshold, which adds 48 to the impact score, above zero and below 100, so orange. The breakdown lists every contribution line by line: every state traces back to a measured reason and a configurable weight.
Volume Outliers¶
Volume detection has always been part of TrackMe — DSM volume outliers, FLX licence-usage use cases. VOL is its standardised, first-class form at scale: one entity per Splunk index, the licensed volume read incrementally from the license usage log with one search over a single source every five minutes, never a 24-hour rescan; ML outliers on drops and spikes, one model per index; trend and month projection. The Global license usage view answers the licence questions people otherwise build dashboards for: who consumes, the trend, days over quota, working-day averages, costs and chargeback. VOL is part of the Foundation edition.
The findings model¶
Some things are not entities with a health score: an account that runs all-time searches, a HEC token gone silent, a receiver pool with a member silent, a fleet of forwarders disconnecting together, a database input that stopped collecting, a Cribl Destination in backpressure, an Edge Fleet left on an old version. For those, seven dedicated components — User Activity Monitoring, Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream and Cribl Edge — share one findings model. A policy evaluates a condition and raises a finding about one object, with its evidence and a severity on the TrackMe scale from Informational to Critical. All findings about one object form a Finding group: one status, one assignee, one severity, one notification thread. Findings recover by themselves when the condition clears; a person decides the rest. The same model brings assignment rules, webhooks and notifications, every component ships its own AI Agent, and nothing is installed on the platforms being watched.
User Activity Monitoring¶
User Activity Monitoring is a dedicated tenant type for one indexing layer seen through one to ten search head tiers. The Findings tab groups the findings about an account and the saved searches it owns into one Finding group; the account view in front shows one person’s activity over the last 24 hours, its executions by search type, run time and the apps used, next to the catalogs of what the account owns.
Sixteen built-in policies in four families — inventory, activity, resources and privileged users — and custom policies in SPL raise the findings; a Privileged users tab shows which accounts hold high privileges and why. UAM can email the account concerned, the UAM AI Agent reviews findings on demand or on a schedule, and nothing is ever changed on the platform.
Splunk HEC¶
A Splunk HEC tenant watches the HTTP Event Collector of one Splunk deployment, Splunk Cloud included, from the telemetry Splunk already writes: every HEC instance logs a heartbeat to _introspection every minute. Endpoints are discovered automatically and grouped into pools, the hosts behind one load balancer, judged as a whole: members up against the expected size, a live balance score, version and configuration drift. Tokens are known by name only, never by value.
Eighteen built-in policies raise findings — availability, senders refused, back pressure naming the pipeline stage that cannot keep up, pool imbalance, token silence — and traffic ML models learn every pool and token. On the left, the pools with one member gone silent; on the right, one endpoint’s ingestion pipeline, every queue with its fill. Nothing is installed on the endpoints.
Splunk S2S¶
A Splunk S2S tenant watches the receiving side of forwarding for one Splunk deployment: the receivers, the intermediate forwarders and the pools they form, from the metrics.log connection lines, queues and throughput, the splunkd.log messages and the health report Splunk already writes. An indexer cluster is a pool, one per site of a multisite cluster; intermediate forwarders are grouped by host name family; senders are counted and typed per pool. On the left, the data path of the last five minutes with the rate on every edge; on the right, Blocked now names the queue that cannot keep up on each host.
Twenty-two built-in policies raise findings: a receiver that stops accepting or goes silent, back pressure and its origin, imbalance, churn, a single destination, TLS and acknowledgement posture. Traffic ML models learn every receiver pool, and the Splunk S2S AI Agent reviews the findings. Nothing is installed on any Splunk instance.
Splunk UF Fleet¶
A Splunk UF Fleet tenant follows the forwarding agents of one Splunk deployment and the Deployment Servers that manage them. An agent is connected when a receiver saw a connection from it within the connected window: the receivers describe the whole fleet, whatever its size, with no heartbeat to deploy. On the left, the fleet now: connected, disconnected and dormant agents, versions, Deployment Servers and the last 24 hours; on the right, a Destination unreachable finding with its sixteen failing senders listed by group.
Mass events are one finding: when many agents disconnect together, one Mass disconnect finding on the fleet or on the group says it. Groups come from the Deployment Servers, server classes, host name patterns, output groups or lookup values; enrichment lookups carry the CMDB onto every agent. Twenty-eight built-in policies cover the agents, the fleet and the Deployment Servers; the Splunk UF Fleet AI Agent reviews the findings.
Splunk Forwarder Collectors¶
A Splunk Forwarder Collectors tenant follows the forwarders a Splunk deployment uses as collection points: heavy forwarders running DB Connect, cloud and API add-ons, scripted and modular inputs, and the forwarders reading syslog files, declared on a group. It reads what Splunk already writes; nothing is installed. On the left, every input as configured, per collector, with its state, the data it delivered and its errors; on the right, one collector opened on its throughput, inputs and errors.
An input is judged by what it delivers, not only by whether it runs. Each feed learns its own rhythm and opens a finding when it stays quiet three times longer than its longest gap ever. Twenty-eight built-in policies cover the collectors, their groups and their inputs; the Splunk Forwarder Collectors AI Agent reviews the findings.
Cribl Stream¶
A Cribl Stream tenant watches one Cribl Stream deployment end to end, from the metrics the Workers already send to a Splunk metric index and from the Leader API, read-only within a shared request budget. On the left, the Worker topology of a self-managed deployment, each Worker with its CPU and memory, live. On the right, a traffic ML model of a Route: what it learned over two weeks, its expected band and the spikes above it.
Sixteen built-in policies raise findings: backpressure, reported health, Pipeline errors, Workers that stop or drift. The Cribl Stream AI Agent reviews findings from checked facts. TrackMe never changes anything in Cribl.
Cribl Edge¶
A Cribl Edge tenant follows the Fleets of one Cribl Edge Leader, the Edge Nodes enrolled in them and the Sources and Destinations they run. It reads the Leader’s node directory in bulk, so its cost does not grow with one call per node and there is no node limit. Fleets are judged as a share of their nodes: one laptop going to sleep opens nothing, a fifth of a Fleet dropping does. Version drift after a rollout, configuration drift after a deploy, failing inputs and outputs and full disks become findings. Any node opens with its system activity read live through the Leader. Edge has no machine learning by design; its AI Agent reviews the findings.
Virtual Tenants¶
A single flat monitoring space does not survive contact with a real Splunk estate. Tenants carve the estate into domains that match how the organisation works. Everything — entities or findings, policies, thresholds, alerts — is scoped to a tenant. An entity tenant enables one or more components: splk-feeds for DSM, DHM and MHM, splk-vol, splk-wlk, splk-flx, splk-fqm. User Activity Monitoring, Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream and Cribl Edge are dedicated tenant types. RBAC is per tenant through Splunk roles, Virtual Groups aggregate several tenants into one read-only card, and Saved Views let a team land on its own tenants.
The monitoring cycle¶
Each cycle, for every tenant: discover, measure, decide, classify, protect, alert. The deliberate separation between what state an entity is in, step 3, and whether to tell you, steps 5 and 6, is why TrackMe is quiet in practice despite tracking so much. Findings tenants run the same idea with their own jobs: collect, evaluate the policies, raise or recover findings, notify. Trackers are scheduled searches, typically every five minutes.
State and impact score¶
Every cycle each entity gets one of four colours. The state derives from a numeric impact score: TrackMe adds up everything currently wrong with an entity into one total score — 0 is green, below 100 orange, 100 and above red. Because every contribution is a number with a configurable weight, you control exactly what makes an entity critical: one major breach trips red, several small ones accumulate. Blue is not a fifth severity — it means “I see this, but I am not waking you up”.
Classification and protection¶
Step 4, classification: priority, tags, labels and SLA are the four kinds of metadata attached to entities. Tags are free-form strings, labels come from a managed coloured catalogue. Three policy families assign them by rule — priority, tags and SLA. Step 5, protection: four mechanisms can hold an entity in blue instead of red. This separation between what state and whether to tell you is deliberate.
Stateful alerting¶
Instead of re-firing every cycle, TrackMe keeps one incident record per problem: it opens when the entity first goes into alert, updates while the problem persists, and closes when the entity recovers. On those transitions it can deliver an email with embedded charts and optionally an AI-written status report, write a notable event, ingest an event and run a command. Findings tenants follow the same principle: every transition of a finding is indexed, and email notifications go one per run or one thread per Finding group.
Topology Studio¶
Topology Studio is the canvas for a view of the environment: a payment flow, a data centre, an ingestion pipeline, a business service. On the left, a map of web services built from TrackMe entities, feeds, hosts and heartbeat checks, with their live state and KPIs. On the right, the flow topology of a Cribl Stream deployment filtered on its nginx Routes, with three Destinations coloured by their findings, ready to save as a live view.
A Splunk HEC tenant generates its own map in one click: pools, endpoints and tokens with live status and sparkline KPIs; a Splunk S2S tenant saves its data path and the Splunk UF Fleet and Splunk Forwarder Collectors tenants their fleet maps the same way. Views are shared under RBAC, aggregates roll many objects into one node, Topology Alerts notify on the health of a whole map, and the AI Assistant can build a view from a sentence. Available in every edition.
The AI layer¶
The AI layer helps operators understand and resolve what TrackMe detects, grounded in the live state of the environment and keeping the operator in the loop. It grew in four levels: the AI Assistant on every page; the AI Advisors and the Concierge, which inspect and, with consent, act; AI Agents automation, running the advisors unattended within guardrails; and AI Routines, your own standing instructions. For findings, the seven dedicated components share the AI Agent, an agentic operator that triages findings, applies the decisions the tenant allows and follows custom instructions. The whole layer is opt-in and inert until an administrator configures a provider.
From a question to a fix¶
A feed is red and the operator asks the AI Assistant why. The Assistant reads the entity, explains the lag and, rather than guessing values, proposes the Feed Lifecycle Advisor with a consent card. The advisor inspects first; on request it acts: it widens the variable delay slots, raises the lag threshold, escalates the priority and documents everything in a note. Every advisor follows the same loop. The ML Advisor, facing a permanent change of baseline, excludes the old period, retrains the model, cancels the stale alert and labels the entity under review: 144 outliers become none. Each action is a typed tool call with its result, audited.
The AI Agent for findings¶
Each of the seven dedicated components, from User Activity Monitoring to Splunk Forwarder Collectors and Cribl Edge, has its own AI Agent on one shared framework. Asked about a finding, or during its hourly scheduled triage, it reads the evidence, the history and the facts about the object, assesses the finding and recommends one decision with its confidence. On the left, the Cribl Stream agent confirms a real health failure on a busy Destination; on the right, the UAM agent confirms oversized CSV lookups on a human account. In act mode it applies only what the tenant allows, resolves only with proof, never overrides an operator, keeps one [AI Agent] note per finding and logs every decision.
Custom instructions, evaluated¶
Administrators give the AI Agent of a findings tenant custom instructions in plain English: what to treat as expected, what to look at first, how to word its recommendations. Every run follows them, asked or scheduled, on every findings tenant, and they never widen what the agent may do. Evaluate runs the agent in inspect mode with the text, saved or not, and shows the same review with and without it, side by side, writing nothing. Here the Cribl Edge agent learns that lab indexers between 50 and 80 percent disk are expected: the same finding reads Expected behaviour instead of a true positive to investigate.
Architecture and editions¶
Three layers: the data plane, trackers and the decision maker or the findings jobs; the control plane, a REST API and per-tenant KV Store; the presentation, a React interface and the AI Assistant. Everything runs inside Splunk, and TrackMe monitors itself: the Configuration Guardian checks its own trackers, connections, indexes and licence and warns before the service degrades. Editions: Foundation covers DSM, DHM, MHM and VOL. Enterprise adds FLX, FQM and WLK and the dedicated components: User Activity Monitoring, Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream and Cribl Edge. Unlimited removes the edition limits; Unlimited Deployments covers every Splunk deployment under one licence. Topology Studio, stateful alerting, policies, the Configuration Guardian, Backup & Restore and the AI Assistant are in every edition. An expired licence first switches to read-only while monitoring keeps running.
The adoption journey¶
This journey is a map, not a ladder: the questions arrive in a particular order and most teams are in several phases at once. Phase 1, data sources with DSM, is where every deployment starts. Phase 2 covers hosts and endpoints with DHM and MHM, usually in a dedicated tenant. Phase 3 is the licensed volume of every index with VOL. Phase 4 is the scheduler with WLK. Phase 5 expands with FLX, FQM and Topology Studio. Phase 6 goes beyond the data: who does what with User Activity Monitoring, the HTTP Event Collector with Splunk HEC, the forwarding path with Splunk S2S and Splunk UF Fleet, the collection points with Splunk Forwarder Collectors, and the Cribl pipelines with Cribl Stream and Cribl Edge. Phase 7, the steady state, is a continuous practice.
Where we are¶
TrackMe is a Splunk app that discovers, maintains and monitors the availability and quality of your data and of the platform around it. Entity monitoring: seven components — DSM, DHM, MHM, VOL, FLX, FQM and WLK — share one entity model, with a state from one impact score, policies that classify, protection that holds noise in blue and stateful alerting. Findings monitoring: seven dedicated components — User Activity Monitoring, Splunk HEC, Splunk S2S, Splunk UF Fleet, Splunk Forwarder Collectors, Cribl Stream and Cribl Edge — raise findings, grouped per object and decided once. Everything is isolated in virtual tenants, with Topology Studio and the AI layer alongside.