Artificial Intelligence in TrackMe

Overview

TrackMe 2.4.0 makes TrackMe Agentic generally available — a family of AI capabilities that help operators investigate, tune, and remediate their Splunk data monitoring directly inside the product. Where TrackMe has always detected data availability and quality problems, the AI layer now helps you understand and resolve them, grounding every answer in the live state of your environment and keeping the operator firmly in the loop.

The AI layer is organised around four surfaces, all built on the same Splunk Agent SDK (the splunklib.ai module shipped with the Splunk Enterprise SDK for Python) runtime, the same provider abstraction, and the same audit trail:

• AI Assistant — the conversational front door. A context-aware chat assistant available on every entity, page and feature of TrackMe (AI Assistant).

• AI Routines — your own standing instructions: user-authored, scheduled, natural-language AI tasks that watch for a condition and then notify you or — with explicit, granular consent — act (AI Routines).

• AI Advisors — five specialists that inspect and remediate a specific monitoring surface (ML models, feed thresholds, FLX thresholds, field-quality dictionaries, component health), plus the AI Concierge, a catalog-driven generalist for the long tail of operations (The AI Advisors, The AI Concierge).

• AI Agents automation — the same advisors running unattended on a nightly schedule, scoped by per-tenant policy (AI Agents automation).

Every action is trustworthy, auditable, and traceable by design: recommendations are typed and grounded in real data, writes are RBAC-checked at the REST boundary, and every change an agent makes is recorded both in the per-entity audit trail and in dedicated audit dashboards (Auditing AI activity).

Important

Opt-in only — The entire AI layer is completely inert until an administrator explicitly configures at least one AI provider. No data is sent to any external service and no LLM calls are made until you opt in. You retain full control over which provider is used and where data is processed — from fully air-gapped on-premises models (Ollama) to Splunk’s own hosted LLMs on Splunk Cloud. See Configuring AI Provider accounts.

Requirements & compatibility

TrackMe remains fully compatible with all supported Splunk releases — the AI layer is additive and opt-in. The Python / Splunk requirements differ by capability:

Capability	Minimum Python	Minimum Splunk
AI Assistant (chat) and AI Concierge	Python 3.9.x	Any Splunk release supported by TrackMe
AI Advisors (ML, Feed Lifecycle, FLX Threshold, FQM, Component Health) and AI Agents automation	Python 3.13.x	Splunk 10.2.x and later (Enterprise or Cloud)
AI Routines (user-authored scheduled AI tasks)	Python 3.13.x	Splunk 10.2.x and later (Enterprise or Cloud)

Important

The AI Advisors are agentic features built on the Splunk Agent SDK, which requires Python 3.13.x — available only from Splunk 10.2.x (Splunk Enterprise or Splunk Cloud). TrackMe enforces this automatically: an Advisor cannot be launched on an incompatible release (a clear message explains the requirement), and the AI Assistant will not propose an Advisor when it detects an incompatible system. The AI Assistant and the AI Concierge run on Python 3.9.x and later, so they remain available on previous Splunk releases.

In short: TrackMe stays fully compatible everywhere; the advanced AI Advisors require Splunk 10.2.x and later. See the compatibility page for the full matrix.

The AI Journey

TrackMe’s AI capabilities were built in layers, and they read best in that order. The AI Assistant was level 1 of the journey — a conversational surface that understands your environment. The Advisors and the Concierge are level 2 — agents that act, with consent. AI Agents automation is level 3 — those same agents operating unattended within guardrails you define. AI Routines are level 4 — you author your own autonomous tasks in natural language: describe what to watch for, and TrackMe evaluates it on your schedule, notifies you, or acts under the consent you granted (AI Routines).

The slides below present the architecture and the core concepts. Each tab focuses on a single idea; the detailed walkthroughs (with product screenshots) live in the dedicated pages of this section.

The big picture

One product, three AI surfaces — each with the operator firmly in the loop.

The AI Assistant is the conversational front door on every entity, tenant and feature page. The AI Advisors are five specialists that inspect and, with consent, act. The AI Concierge is the catalog-driven generalist for everything else — backed by the live TrackMe REST API. Same Splunk Agent SDK underneath, same provider abstraction, same audit surface.

What the Advisors do

Four capabilities, one autonomous loop — applied across DSM, DHM, FLX, WLK/MHM and FQM.

Every advisor follows the same loop: Observe (read entity context, metric history, scoring, audit trail), Diagnose (classify the evidence against a typed set of scenarios — no free-text guesses), Recommend (produce structured, reviewable recommendations), and — in act mode — Remediate (apply the fix through TrackMe’s own REST API, recording every write to the audit trail).

Six advisors

Five specialists plus the Concierge.

Each specialist owns a curated tool registry scoped to its monitoring surface — ML Advisor (outlier model integrity), Feed Lifecycle (DSM/DHM thresholds), FLX Threshold (FLX dynamic thresholds), Component Health (WLK + MHM), and FQM Advisor (field quality and data dictionary). The AI Concierge is catalog-driven and consent-gated. They share one orchestration library, the same MCP tool pattern, and the same Pydantic output contract.

Architecture

One shared control plane.

The Assistant and the chat bridge sit on the entry side; the Advisors and the Concierge dispatch through the same SDK runtime. A REST handler launches a daemon thread that runs the splunklib.ai.Agent (LangGraph) loop against either MCP tools (specialists) or the live API catalog (Concierge). RBAC is enforced at every REST boundary, concurrency is capped, every run is bounded with retries, and every outcome is audited end-to-end.

Built on the Agent SDK

We stand on the SDK Splunk publishes, and share it across every AI surface.

The advisors are built on the Splunk Agent SDK (splunklib.ai.Agent) rather than direct LLM API calls: it ships the LangGraph orchestration engine, the MCP tool registry, structured output via Pydantic, and middleware hooks for tracing and limits. The provider abstraction means the same advisor code runs across OpenAI, Azure OpenAI, Anthropic, Google, Mistral, Ollama, OpenAI-compatible endpoints and Splunk Hosted — no provider lock-in.

Inspect or Act

Tag-filtered tool sets, same binary — a different toolbox presented to the LLM.

Every specialised advisor honours a two-phase contract. Inspect mode is read-only and always safe to run: 5–7 read tools, a smaller token/step budget, structured recommendations only, no writes — the write tools are literally absent from the agent’s toolbox. Act mode adds the write tools (11–18), a larger budget, and a hard rule that every write is verified by a follow-up read; each write is prefixed [AI Agent] and lands in the audit trail.

The Assistant ↔ Advisor bridge

How the chat learns about the advisors, sees recent runs, and proposes a one-click consent card.

Each describe payload carries two channels: static knowledge_reference (what each advisor is, its modes, gates and contract) and dynamic ai_advisor_recent_runs (RBAC-filtered recent runs). When the user describes a symptom, the assistant emits a typed advisor_invocation contract that the chat renders as a consent card. The chat never directly calls a write tool — it always proposes an advisor (or Concierge) invocation, and the user’s click is the structural safety property.

The AI Concierge

Catalog-grounded, consent-gated, zero static write tools — the long tail without the foot-guns.

The Concierge grounds itself in the live TrackMe REST API catalog (~423 endpoints) and proposes structured action contracts that the chat consent card renders for explicit, per-action approval. It has zero static write tools by design: every proposed path is validated against the live catalog, destructive actions require typed confirmation, and the user’s confirmation click is what fires the REST call — through their own credentials, RBAC-checked.

Structured output & guardrails

Three pillars — Trust, Auditability, Traceability — built into the agent runtime, not bolted on after.

Trust: typed decisions, not free text — every agent emits a Pydantic envelope (entity_status / summary / recommendations / actions_taken / reasoning_trace); invalid completions are retried or rejected by the SDK. Auditability: runtime guardrails enforce the blast radius — tag-filtered write gate, decommission guard, per-mode tool selection. Traceability: every step is indexed and queryable, and every entity write carries the [AI Agent] prefix so a human can see exactly what changed and why.

Governance

One menu in TrackMe consolidates AI configuration and observability for operators.

The Artificial Intelligence menu brings together everything an operator needs: Manage AI providers (create and test provider accounts), Manage AI tenant configuration (per-tenant master switches, per-advisor enablement, the decommission guard — disabled by default everywhere), and the two audit dashboards for AI Advisor and AI Assistant activity. Operators own the controls, everything is disabled by default, and all activity is indexed to the summary index and queryable in plain SPL.

The three pillars

Every design decision across the AI layer traces back to three principles:

Pillar	What it means in practice
Trust	The agent never makes up entity state. Every claim traces to a tool call, and the system prompts forbid speculation. Outputs are typed — a Pydantic schema forces a structured envelope rather than free-form prose, and invalid completions are retried or rejected by the SDK. Read-only inspect mode is the default for unfamiliar entities.
Auditability	Runtime guardrails — not soft prompts — enforce the blast radius: a tag-filtered write gate (write tools are absent in inspect mode), a decommission guard (disabled by default), and per-mode tool selection. Hard runtime caps (token, step, and wall-clock budgets) are enforced by SDK middleware and a watchdog, so a misbehaving run hits a real wall.
Traceability	Every tool call and every advisor run is indexed to the tenant summary index (trackme:ai_agent:*), and every entity write carries an [AI Agent] <reason> prefix in the per-entity audit trail. You can answer “which conversation triggered this change, and why?” with a single SPL query.

What’s in this section

This Artificial Intelligence section walks the full journey, with product screenshots at each step:

Page	What it covers
Configuring AI Provider accounts	Configuring one or more AI provider accounts (OpenAI, Azure, Anthropic, Gemini, Mistral, Ollama, Custom, Splunk Hosted) and the global AI settings. Start here.
AI Assistant	The conversational AI Assistant — capabilities, opening it from any page, the chat interface, context enrichment, AI status reports in alert emails, and troubleshooting.
AI Routines	AI Routines — describe a monitoring task in plain English; TrackMe watches for it on a schedule, then notifies you or (with consent) acts. Templates, the Evaluate dry-run gate, permission families, cost transparency, and auditing.
The AI Concierge	The AI Concierge — the catalog-driven generalist reached from the Assistant for operational requests, with consent cards and validation gates.
The AI Advisors	The AI Advisor family — inspect/act, the consent-card bridge, and the FLX Threshold Advisor.
AI Feeds Advisor	The AI Feeds Advisor (Feed Lifecycle) — DSM/DHM threshold tuning.
AI ML Advisor	The AI ML Advisor — outlier model inspection and remediation.
AI FQM Advisor	The AI FQM Advisor — field-quality thresholds and data-dictionary management.
AI Component Health Advisor	The AI Component Health Advisor — WLK and MHM health.
AI Agents automation	AI Agents automation — running the advisors unattended on a nightly schedule, scoped by per-tenant policy.
Auditing AI activity	Auditing AI activity — the AI Advisor and AI Assistant audit dashboards.
AI Compliance, Privacy and Data Governance	AI compliance, privacy and data governance — the reference for compliance, security and privacy teams: data flows, what is (and is not) sent to the LLM, the complete tool inventory, guardrails, RBAC, audit, and egress requirements.