AI Assistant

Overview

The TrackMe AI Assistant is an integrated, context-aware AI chat assistant available across the entire TrackMe product. It helps users investigate entity health issues, manage operational workflows, and get instant guidance on any TrackMe feature — all directly within the Splunk interface.

Note

Compatibility — The AI Assistant is compatible with Python 3.9.x and later, so it is available on the same Splunk releases as the rest of TrackMe. The advanced AI Advisors additionally require Splunk 10.2.x and later (Python 3.13.x); see Artificial Intelligence in TrackMe (Requirements & compatibility) and the compatibility page.

Capabilities

Three pillars of AI-powered assistance across TrackMe.

The AI Assistant covers entity investigation, operational guidance across all TrackMe features, and AI-powered email notifications — all from a single, context-aware chat interface.

Entity Investigation

Open the AI Assistant from any entity panel to get instant, context-aware analysis.

The assistant explains why an entity is in alert state, analyses lag metrics, event counts, and ML results, provides actionable SPL investigation searches, and suggests threshold adjustments with full context. Covers DSM, DHM, MHM, FLX, FQM, and WLK entities.

Context Is Everything

The real power of TrackMe’s AI isn’t the model — it’s the context.

TrackMe’s describe API endpoints feed rich, structured data to the LLM, so it understands the exact state of your environment without the user having to explain anything. Context types include entity investigation, virtual tenants, REST API reference, backup & restore, maintenance mode, and license management.

Context-Aware Intelligence

The AI automatically collects structured data from the page you’re on — no need to describe the situation.

Suggested questions adapt to each page context, and the assistant provides targeted analysis whether you’re investigating an entity alert, exploring ML Outliers anomalies, or navigating the REST API.

Provider Agnostic

Connect to the LLM provider that fits your organisation.

Configure multiple providers for comparison, cost optimisation, or redundancy. Supported providers include OpenAI, Azure OpenAI, Anthropic, Google Gemini, Mistral AI, Ollama, Splunk Hosted (SLIM API), and any OpenAI-compatible endpoint.

Security & Privacy

Security and privacy by design — full control over your AI infrastructure.

Opt-in only (completely inert until configured), RBAC enforced (users only access data they’re authorised to view), credential isolation (API keys stored encrypted, never exposed to end users), and optional data anonymisation (SHA256 hashing of entity and index names).

Email Notifications

Stateful alert emails can include an AI-generated status summary, adapting to the incident lifecycle.

The AI report explains what went wrong on incident open, analyses changes on update, and confirms recovery on close. Uses the same provider configuration, operates fail-open (never blocks email delivery), and is capped at 120 seconds to avoid delays.

Architecture

Asynchronous, streaming architecture with built-in security controls.

The flow proceeds in four steps: user opens chat from any TrackMe page, describe API collects structured entity data, a background thread streams the LLM call, and tokens arrive in real-time to the UI. RBAC enforcement, concurrency control, encrypted credentials, and automatic session cleanup are built in.

Important

Opt-in only — The AI Assistant is entirely opt-in. While the feature toggle is enabled by default so that administrators can discover it, no AI functionality is active, no data is sent to any external service, and no LLM calls are made until an administrator explicitly configures at least one AI provider. Without a configured provider, the feature is completely inert.

Tip

Full control over your AI infrastructure — Organisations with strict data privacy, data sovereignty, or regulatory requirements retain complete control over which AI provider is used and where data is processed. TrackMe can be connected to:

• Splunk’s own hosted LLMs via the Splunk Hosted (SLIM API) provider type on Splunk Cloud, with zero configuration beyond selecting the provider and model — no API keys, no base URL, fully managed by the Splunk platform

• Self-hosted open-source LLMs via Ollama, keeping all data entirely on-premises with zero external API calls

• Private or corporate LLM deployments via the Custom (OpenAI-compatible) provider type, enabling integration with any internally hosted endpoint that implements the OpenAI chat completions format

• Cloud providers of your choice, including region-specific deployments (e.g. Azure OpenAI in a specific geography) to meet data residency requirements

The choice of provider is yours — from fully air-gapped on-premises models to managed cloud services — and no data is ever shared between configured providers.

Configuration

Before the AI Assistant can be used, an administrator must configure at least one AI provider account and (optionally) review the global AI settings. Both are covered in their own page:

• Configuring AI Provider accounts — create and test provider accounts (OpenAI, Azure OpenAI, Anthropic, Google Gemini, Mistral AI, Ollama, Custom, Splunk Hosted), and the global Enable AI assistant, concurrency, and anonymisation settings.

Important

At least one AI provider account must be configured for the AI Assistant to function. If no provider is configured, users see an informative error message when they attempt to open the AI chat panel. See Configuring AI Provider accounts.

Usage

Opening the AI Assistant

The AI Assistant is available throughout TrackMe via two patterns depending on the context.

From Entity Investigation

When investigating a specific entity (DSM, DHM, MHM, FLX, FQM, WLK), the AI Assistant can be opened via:

Entity overview panel:

1. Open an entity investigation panel by clicking on an entity in the monitoring dashboard

2. Click the AI Assistant button in the panel header

3. The AI chat panel opens as a full-screen slide panel with the entity context pre-loaded

Entity actions menu:

1. Click the three-dot actions menu (kebab menu) on any entity row in the monitoring dashboard

2. Select Ask AI

3. The AI chat panel opens directly with the entity context pre-loaded

From TrackMe UI Pages

Each TrackMe UI page features an AI assistant button in the top-right header. Clicking it opens the AI chat panel with the relevant context for that page automatically loaded.

The AI Assistant is available from the following pages:

• Virtual Tenants — context includes all tenants, their health states, and configuration

• Tenant Home — context includes tenant-specific entities and component overview

• REST API Reference — context includes the full API catalogue, resource groups, and usage patterns

• Backup & Restore — context includes backup records, status, and scheduling information

• Global Maintenance Mode — context includes current maintenance status, scheduling, and countdown

• Maintenance Knowledge Database — context includes KDB records and per-entity maintenance windows

• Bank Holidays Management — context includes holiday periods, country coverage, and recurring calendars

• License Management — context includes edition, validity, expiration, and feature information

Note

The AI assistant button is only visible when the feature is enabled in the General settings and at least one AI provider is configured.

Chat Interface

The AI chat panel provides a conversational interface with the following features:

Suggested questions: When the chat panel opens, context-specific suggested questions are displayed to help get started. The questions are tailored to the page the assistant was opened from. For example, entity investigation suggests questions about health state and thresholds, while the Backup & Restore page suggests questions about backup status and restore procedures.

Free-form questions: Type any question in the input field at the bottom of the panel. Press Enter to send (use Shift+Enter for a newline).

Streaming responses: AI responses stream in real-time as tokens are generated, providing immediate feedback.

Conversation context: The AI Assistant maintains conversation history within the session. Follow-up questions are understood in the context of previous messages.

Markdown rendering: AI responses support full Markdown formatting including code blocks, tables, lists, and inline formatting. SPL searches are displayed in syntax-highlighted code blocks for easy copy-paste.

Provider switching: If multiple providers are configured, use the dropdown at the top of the chat panel to switch between providers. Switching providers clears the conversation history.

Clear chat: Use the trash icon button to clear the conversation and start fresh.

Worked examples

The simplest way to experience the AI Assistant is to open it from an entity and ask a plain question. In the example below, the assistant is opened from Tenant Home and then on a specific DSM entity; the suggested questions adapt to the entity, and a question such as “What is the difference between latency and delay in TrackMe? How is it calculated?” is answered with a precise, context-aware explanation grounded in that entity’s live metrics.

Because the assistant has the entity’s full context, it can also produce ready-to-run SPL searches tailored to the question — for example when asking “What search can I use to investigate latency issues?”. SPL is rendered in syntax-highlighted code blocks with Copy to clipboard and Execute in Search buttons, so you can move from question to investigation in one click.

Tip

The AI Assistant is the front door to the wider AI layer. When you describe a problem it can resolve, the assistant proposes a one-click consent card that hands off to a specialist AI Advisor or to the AI Concierge — see those pages for the full agentic workflow.

Context Enrichment

When a chat session starts, TrackMe automatically collects structured data relevant to the page the assistant was opened from. This context is passed to the LLM as part of a specialised system prompt, enabling the AI to provide specific, data-driven answers without the user having to describe the situation manually.

The following table summarises what context is collected for each area:

Context	Data Collected
Entity investigation	Health state, scoring, data lag and event count metrics, configured thresholds, ML Outliers detection results, investigation SPL searches, health history, entity metadata (index, sourcetype, host, etc.), and threshold tuning guidance.
Virtual Tenants	All tenants with their health states, component counts, configuration, and license information.
Tenant Home	Tenant-specific entities, health overview, components, and configuration details.
REST API Reference	Full API catalogue including resource groups, endpoints, HTTP methods, authentication patterns, SPL | trackme command syntax, and the describe=true self-documentation pattern.
Backup & Restore	Backup archive records (timestamps, labels, status, type), restore history, and a knowledge reference covering backup procedures and best practices.
Global Maintenance Mode	Current maintenance status (enabled/disabled/scheduled), start and end times, countdown to auto-disable, and a knowledge reference on maintenance impact.
Maintenance Knowledge Database	KDB record counts, recent records sorted by end time (planned and unplanned types), per-entity maintenance windows, and guidance on KDB management.
Bank Holidays Management	Holiday periods, country coverage, recurring calendar configuration, and information on how holidays affect SLA calculations and monitoring thresholds.
License Management	Current edition, validity and expiration status, feature availability per edition, developer mode status, and renewal workflow guidance.

Entity Threshold Tuning Guidance

The entity investigation context includes a thresholds_help section that provides entity-type-specific advisory guidance for threshold tuning. This enables the AI Assistant to give accurate, actionable advice when users ask about adjusting thresholds:

• DSM / DHM: Explains event delay, ingestion latency, future tolerance, adaptive delay, variable delay, and lagging class thresholds. DSM entities additionally include host count monitoring guidance.

• MHM: Covers metric lag and future tolerance thresholds, as well as the metric category policy.

• FLX: Distinguishes between entity-level settings (inactivity timeout) and use case-level threshold conditions defined in the FLX tracker configuration.

• FQM: Explains that thresholds are managed at the FQM tracker level (success rate and coverage conditions).

• WLK: Explicitly notes that Workload Monitoring entities do not have user-configurable thresholds, and advises investigating execution issues instead.

Session Lifecycle

Panel close and job cancellation: When the AI chat panel is closed (or the user navigates away), the frontend automatically cancels any in-flight AI request. This releases the concurrency slot immediately so other users can use the AI Assistant without hitting the concurrency limit. The backend worker thread may continue briefly until the LLM responds, but the response is discarded.

Custom Prompt

Administrators can use the Custom Prompt field in the AI Provider configuration to add domain-specific instructions that are appended to the system prompt. Example use cases:

• Enforce response formatting (e.g. always include a “Next Steps” section)

• Add organisation-specific context (e.g. known maintenance windows, team ownership)

• Customise the assistant’s tone or verbosity

• Include relevant runbook references or escalation procedures

AI Status Report in Stateful Alert Emails

In addition to the interactive chat assistant, the AI infrastructure powers an optional AI Status Report feature in stateful alert email notifications. When enabled, each email notification includes an AI-generated entity status summary embedded directly in the HTML email body.

How it works:

1. When a stateful alert triggers and email delivery is required, TrackMe calls the configured AI provider with the entity context (the same describe data used by the interactive chat).

2. The AI generates a concise, actionable summary tailored to the incident lifecycle stage:

   • Opened: Explains what went wrong and where to investigate first

   • Updated: Analyses whether the situation has changed since the last notification

   • Closed: Confirms the recovery and summarises what was previously wrong

3. The AI response is converted to email-safe HTML and embedded in the notification between the detailed information and charts sections.

Key design principles:

• Same provider configuration: The AI Status Report uses the same AI provider(s) configured in Configuration > AI Provider. No additional AI setup is required.

• Fail-open: If the AI provider is unavailable, slow, or returns an error, the email notification is delivered normally without the AI section. No email is ever blocked by an AI failure.

• Cost-aware: The AI report is only generated when email delivery is actually needed. If the entity’s priority level is filtered out from email notifications by priority routing, no AI call is made.

• Timeout capped: The AI request timeout is capped at 120 seconds for email flows to avoid excessive notification delays.

Example of an email notification with the AI Status Report:

Configuration:

The AI Status Report is configured per stateful alert in the alert creation wizard or in the Splunk alert action editor:

• AI Status Report: Enable or disable. Defaults to enabled when at least one AI provider is configured.

• AI Provider: Optionally select a specific provider when multiple are configured. Defaults to the first available provider.

For detailed setup instructions, see Stateful alerting.

Troubleshooting:

AI Status Report operations are logged with the task=ai_status_report tag in the alert action logs:

index=cim_modactions sourcetype=modular_alerts:trackme_stateful_alert "task=ai_status_report"

Error Handling

The AI Assistant handles errors gracefully with clear user-facing messages:

Error Type	Description
AI not configured	No AI provider has been configured. An administrator must set up at least one provider in the Configuration page.
AI disabled	The AI assistant has been disabled by an administrator in the General settings.
AI busy	The maximum number of concurrent AI chat requests has been reached. The user should try again in a moment.
API key missing	The AI provider requires an API key but none is configured. This can occur if a provider was created without saving the API key, or if the credential was lost. Re-enter the API key in the AI Provider configuration.
Provider error	The AI provider returned an error (e.g. invalid API key, model not found, rate limit exceeded). The error message from the provider is displayed.

Troubleshooting

Logs

The AI Assistant logs all requests, errors, and streaming activity to dedicated log files on the Splunk search head. Use the following SPL search to access AI-related logs:

index=_internal sourcetype="trackme:rest_api" source="/opt/splunk/var/log/splunk/trackme_rest_api_ai_*"

Key log events to look for:

Log Pattern	Description
action="request_started"	A new AI chat request was submitted. Includes the provider, model, base URL, timeout, context type, and whether context was loaded.
action="first_token"	The first token was received from the LLM provider. The time_to_first_token_sec field indicates provider responsiveness.
action="streaming_progress"	Periodic heartbeat during long-running streaming responses (every 30 seconds). Shows elapsed time and characters received.
action="timeout"	The LLM request timed out. Check the timeout_sec and elapsed_sec fields. Consider increasing the Request Timeout setting.
action="stale_job_cleanup"	A running job was detected as stale (exceeded its timeout plus a grace period). The concurrency slot was released automatically.
action="slot_released"	A concurrency slot was released, either after job completion or via panel close cancellation.
exception=	An error occurred during the AI request. The exception message typically includes the HTTP status code and error body from the provider.

Common Issues

“Missing Authorization header” from Google Gemini

The API key is not configured or was not saved correctly for this provider. Edit the provider in Configuration > AI Provider, re-enter the API key, and save.

“Model not found” (HTTP 404)

The model name is incorrect. Verify the exact model ID in your provider’s documentation. Common mistakes include missing version suffixes (e.g. gemini-3-pro vs gemini-3-pro-preview) or using aliases that don’t exist for the provider.

“AI assistant is at maximum capacity”

All concurrency slots are in use. Either wait for an in-flight request to complete, or increase the Max concurrent AI chats setting in Configuration > General. If slots appear stuck, check for stale jobs in the logs — they are automatically cleaned up after the configured timeout plus a grace period.

Streaming responses appear slow or stalled

For cloud providers, this is typically caused by network latency or provider-side rate limiting. For Ollama, ensure the server has sufficient resources (CPU/GPU/RAM) for the model. Check the time_to_first_token_sec in the logs to distinguish between slow initial response and slow token generation.

Context not loaded

If the AI responds with generic advice rather than specific analysis, check the entity_context_loaded field in the request log. A value of False indicates the context collection failed. For entity investigations, verify that the entity exists in the specified tenant and that the user has RBAC permissions to access it. For other contexts, check that the relevant TrackMe backend services are running correctly.