The AI Advisors

Overview

The AI Advisors are TrackMe’s agentic surface — specialists that read the same KV Store and search indexes a human analyst would, propose remediations grounded in real configuration and recent telemetry, and (in act mode) apply changes through the same REST API a UI click would invoke. Every change is auditable, every write is RBAC-checked at the REST boundary, and every recommendation can be replayed.

There are six advisors — five specialists plus the catalog-driven AI Concierge:

Advisor	What it handles	Documentation
AI Feeds Advisor (Feed Lifecycle)	DSM / DHM threshold tuning, monitoring state, adaptive delay, variable-delay schedules, impact-score weights	AI Feeds Advisor
AI ML Advisor	ML outlier model inspection and remediation (period exclusions, retrains, density-threshold tuning, false-positive clears)	AI ML Advisor
AI FQM Advisor	Field-quality thresholds and data-dictionary management (per-field validation, CIM compliance, regex testing); wizard-time dictionary generation	AI FQM Advisor
AI Component Health Advisor	WLK (workload scheduled-search health) + MHM (metric-host lag) thresholds	AI Component Health Advisor
AI FLX Threshold Advisor	FLX dynamic thresholds (add / update / delete / variable-threshold slots), entity state and priority	(below)
AI Concierge	Long-tail operational requests grounded in the live REST API catalog	The AI Concierge

Note

The advisors are opt-in: they require at least one configured AI provider account, and each is disabled by default until enabled per tenant (see AI Agents automation for the automated path). On-demand interactive runs from the UI and the chat are available to authorised operators as soon as a provider is configured.

Important

The AI Advisors require Splunk 10.2.x and later (Python 3.13.x). The five specialised advisors are built on the Splunk Agent SDK, which needs Python 3.13.x — shipped only with Splunk 10.2.x (Splunk Enterprise or Splunk Cloud). TrackMe enforces this automatically: an advisor cannot be invoked on an incompatible release (a clear message explains the requirement), and the AI Assistant will not propose an advisor when it detects an incompatible system. The AI Assistant and the AI Concierge do not require Python 3.13.x — they run on Python 3.9.x and later, so they remain available on previous Splunk releases. See Artificial Intelligence in TrackMe (Requirements & compatibility) and the compatibility page.

Inspect or Act — the operational contract

Every specialised advisor honours the same two-phase contract:

Mode	Behaviour
inspect	Read-only — always safe to run. The agent analyses and surfaces structured recommendations, then stops. No KV writes, no REST mutations: the write tools are literally absent from the agent’s toolbox. This is the default for unfamiliar entities and the first stage of the chat bridge.
act	Full remediation. The agent applies the recommended changes through write tools, then re-reads to confirm. A hard rule requires every write to be verified by a follow-up read, and every write is prefixed [AI Agent] <reason> in the entity audit trail.

The two phases give the chat bridge a natural two-step flow: the operator describes the problem → the assistant proposes an inspect run → the operator reviews the recommendations and clicks Run Act (optionally with extra instructions) to apply them.

Note

Hard runtime caps are enforced by SDK middleware and a shared watchdog, not by prompt text — an inspect run has a smaller token / step budget and a 10-minute wall-clock cap; an act run has a larger budget and a 15-minute cap. A misbehaving run hits a real wall and is recorded as an error.

Two ways to run an advisor

Standalone modal

Every entity overview exposes an advisor tab appropriate to its component (for example AI Feeds Advisor on DSM/DHM, AI ML Advisor on the Outliers tab, AI FQM Advisor on FQM, AI WLK Advisor on WLK). The modal lets you pick the provider and mode, add optional instructions, and launch the run — showing a live tool-call progress feed and then the structured result with status badge, recommendations, actions taken, and a collapsible reasoning trace.

The chat consent-card bridge

The AI Assistant knows about the advisors. When you describe a symptom an advisor can resolve, it proposes a typed advisor invocation rendered as an inline consent card — without leaving the conversation. The chat never calls a write tool directly; it always proposes an advisor (or Concierge) invocation, and your click is the structural safety property.

The example below walks the full bridge flow on a DSM entity. The operator opens the assistant on the entity and asks the assistant to “analyse and design proper settings for this feed”.

The assistant reasons over the entity’s live context and recommends the Feed Lifecycle Advisor, launching it in inspect mode. A live tool-call feed shows each read the agent performs (get_entity_lifecycle_context, get_entity_delay_latency_history, get_entity_variable_delay_schedule, …).

The inspect result is rendered as structured recommendations (here a variable-delay schedule and a priority change) each with current value, recommended value and rationale, plus a collapsible reasoning trace. From the result, a Run Act now panel offers an optional additional instructions textbox before applying the changes.

When you run act, the advisor applies the changes through TrackMe’s REST API and verifies each write. It also attaches a formatted investigation & remediation report as an entity note, prefixed [AI Agent] — documenting the root-cause analysis, the entity summary, and every change applied — so the full story is visible directly on the entity.

Structured output and guardrails

Every advisor run emits a typed envelope (validated by the SDK) rather than free-form prose:

• entity_status — an at-a-glance verdict (e.g. healthy / needs_tuning / decommission_candidate).

• summary — a short executive summary.

• recommendations — structured items, each with the field, current value, recommended value and a rationale that cites the tool calls that produced it.

• actions_taken — populated in act mode only, one entry per write that actually occurred.

• reasoning_trace — a step-by-step trace citing the read (and write) tool calls.

The guardrails that bound every run:

• Ground every claim in data — the agent’s first calls are always reads; a run with zero read tool calls is a bug. Speculation is forbidden by the system prompts.

• Tag-filtered write gate — write tools are present only in act mode.

• Decommission guard — disabling an entity (monitored_state=disabled) is gated and disabled by default in the automated path.

• RBAC at the REST boundary — the agent inherits the operator’s session; an unauthorised write fails cleanly with a 403.

• Audit reason discipline — every write requires a meaningful reason citing the field, the from/to values, and the operational trigger; trivial reasons are rejected.

The AI FLX Threshold Advisor

The FLX Threshold Advisor handles FLX (Flex Objects) — the “any SPL search as a tracked entity” component. FLX entities expose custom KPIs (a metrics JSON blob computed by the tracker’s search), dynamic thresholds (per-metric value comparisons with configurable operators), and optional variable thresholds (time-slot-based values for day-of-week / hour-of-day variation). The advisor answers: “are these thresholds tuned for this entity’s actual behaviour, or did the wizard’s defaults drift?”

When to use it

• A FLX entity is RED and you want to know whether the threshold was wrong (over-tight) or the data is genuinely anomalous.

• A new FLX entity created via the wizard has been running for a week and you want to validate the applied defaults against observed behaviour.

• A metric’s breach pattern matches business-hours / off-hours variation and a variable-threshold schedule would fit better than a static one.

• You want to delete obsolete thresholds or review a customised use-case template entity.

What it reads and writes

In inspect mode the advisor reads the full FLX entity context, the 24h + 7d metric value history, the threshold-breach history, the entity’s use-case template definition, and peer-entity thresholds. In act mode it can add / update / delete dynamic thresholds, configure variable-threshold slots, and update the entity’s monitoring state or priority (decommission-gated).

Note

The FLX Threshold Advisor is the most hardened advisor against the FLX condition_true flag, which inverts alert semantics. Inverse-style (condition_true=true) means “alert fires when the comparison is FALSE” — used for “healthy if X holds” tracking. Direct-style (condition_true=false) means “alert fires when the comparison is TRUE” — used for “alert if X holds” tracking. The advisor validates a proposed threshold’s direction before recommending it.

Where to find it

The AI Threshold Advisor tab appears on FLX entity overview modals, and the chat bridge proposes it for FLX threshold intents. It participates in the nightly automated batch — see AI Agents automation.

Tip

Beyond tuning existing entities, this advisor can also build a brand-new FLX tracker from a plain-language description at wizard time — grounded on the use-case library and validated by a live dry-run. See Natural-language FLX tracker generation.

Related

• AI Feeds Advisor, AI ML Advisor, AI FQM Advisor, AI Component Health Advisor — the per-advisor walkthroughs.

• The AI Concierge — the catalog-driven generalist for everything outside the specialists.

• AI Agents automation — running the advisors unattended on a schedule.

• Auditing AI activity — auditing every advisor run.

• AI Feeds Advisor
• AI ML Advisor
• AI FQM Advisor
• AI Component Health Advisor