AI Assistant¶

Overview¶

The TrackMe AI Assistant is an integrated, context-aware AI chat assistant available across the entire TrackMe product. It helps users investigate entity health issues, manage operational workflows, and get instant guidance on any TrackMe feature — all directly within the Splunk interface.

Capabilities

Three pillars of AI-powered assistance across TrackMe.

The AI Assistant covers entity investigation, operational guidance across all TrackMe features, and AI-powered email notifications — all from a single, context-aware chat interface.

Entity Investigation

Open the AI Assistant from any entity panel to get instant, context-aware analysis.

The assistant explains why an entity is in alert state, analyses lag metrics, event counts, and ML results, provides actionable SPL investigation searches, and suggests threshold adjustments with full context. Covers DSM, DHM, MHM, FLX, FQM, and WLK entities.

Context Is Everything

The real power of TrackMe’s AI isn’t the model — it’s the context.

TrackMe’s describe API endpoints feed rich, structured data to the LLM, so it understands the exact state of your environment without the user having to explain anything. Context types include entity investigation, virtual tenants, REST API reference, backup & restore, maintenance mode, and license management.

Context-Aware Intelligence

The AI automatically collects structured data from the page you’re on — no need to describe the situation.

Suggested questions adapt to each page context, and the assistant provides targeted analysis whether you’re investigating an entity alert, exploring ML Outliers anomalies, or navigating the REST API.

Provider Agnostic

Connect to the LLM provider that fits your organisation.

Configure multiple providers for comparison, cost optimisation, or redundancy. Supported providers include OpenAI, Azure OpenAI, Anthropic, Google Gemini, Mistral AI, Ollama, Splunk Hosted (SLIM API), and any OpenAI-compatible endpoint.

Security & Privacy

Security and privacy by design — full control over your AI infrastructure.

Opt-in only (completely inert until configured), RBAC enforced (users only access data they’re authorised to view), credential isolation (API keys stored encrypted, never exposed to end users), and optional data anonymisation (SHA256 hashing of entity and index names).

Email Notifications

Stateful alert emails can include an AI-generated status summary, adapting to the incident lifecycle.

The AI report explains what went wrong on incident open, analyses changes on update, and confirms recovery on close. Uses the same provider configuration, operates fail-open (never blocks email delivery), and is capped at 120 seconds to avoid delays.

Architecture

Asynchronous, streaming architecture with built-in security controls.

The flow proceeds in four steps: user opens chat from any TrackMe page, describe API collects structured entity data, a background thread streams the LLM call, and tokens arrive in real-time to the UI. RBAC enforcement, concurrency control, encrypted credentials, and automatic session cleanup are built in.

Important

Opt-in only — The AI Assistant is entirely opt-in. While the feature toggle is enabled by default so that administrators can discover it, no AI functionality is active, no data is sent to any external service, and no LLM calls are made until an administrator explicitly configures at least one AI provider. Without a configured provider, the feature is completely inert.

Tip

Full control over your AI infrastructure — Organisations with strict data privacy, data sovereignty, or regulatory requirements retain complete control over which AI provider is used and where data is processed. TrackMe can be connected to:

Splunk’s own hosted LLMs via the Splunk Hosted (SLIM API) provider type on Splunk Cloud, with zero configuration beyond selecting the provider and model — no API keys, no base URL, fully managed by the Splunk platform
Self-hosted open-source LLMs via Ollama, keeping all data entirely on-premises with zero external API calls
Private or corporate LLM deployments via the Custom (OpenAI-compatible) provider type, enabling integration with any internally hosted endpoint that implements the OpenAI chat completions format
Cloud providers of your choice, including region-specific deployments (e.g. Azure OpenAI in a specific geography) to meet data residency requirements

The choice of provider is yours — from fully air-gapped on-premises models to managed cloud services — and no data is ever shared between configured providers.

Configuration¶

The AI Assistant configuration is split across two areas in the TrackMe Configuration page:

Configuration > Artificial Intelligence (General settings tab): Global enable/disable toggle, concurrency limit, and anonymization options
Configuration > AI Provider (dedicated tab): Provider-specific configuration (one or more providers)

General Settings¶

Navigate to Configuration > General and locate the Artificial Intelligence group.

Setting	Default	Description
Enable AI assistant	Yes	Enable or disable the AI assistant feature across all TrackMe UIs. When disabled, the AI assistant button is hidden from all users.
Max concurrent AI chats	10	Maximum number of concurrent AI chat requests across all users. Additional requests are rejected with an HTTP 429 response until a slot becomes available.
Anonymize entity names for AI	No	When enabled, entity names (object and alias fields) are anonymized using SHA256 hashing in describe endpoint responses consumed by AI providers. This protects potentially sensitive infrastructure details (hostnames, IPs, machine names) while preserving the object_id for entity identification. The AI assistant is guided to use object_id references instead of entity names.
Anonymize index names for AI	No	When enabled, Splunk index names are anonymized using SHA256 hashing in describe endpoint responses consumed by AI providers. This applies to data source (DSM), data host (DHM), and metric host (MHM) entities. Index names are replaced with their SHA256 hashes in entity information, investigation searches, and metric details. Investigation searches will contain hashed index names that users must replace with actual values before execution. The AI assistant is guided to inform users that index references in searches are anonymized placeholders.

AI Provider Configuration¶

Navigate to Configuration > AI Provider to create one or more provider configurations. Each provider configuration defines a connection to an LLM endpoint.

Important

At least one AI provider must be configured for the AI Assistant to function. If no provider is configured, users see an informative error message when they attempt to open the AI chat panel.

Provider Fields¶

Field	Default	Description
AI Provider	OpenAI	The AI provider type. Supported values: `OpenAI`, `Azure OpenAI`, `Anthropic`, `Google Gemini`, `Mistral AI`, `Ollama`, `Custom (OpenAI-compatible)`, `Splunk Hosted (SLIM API)`.
Base URL	(optional)	The base URL for the AI provider API. Required for all providers except Splunk Hosted, which auto-discovers its endpoint from the Splunk Cloud platform. See the Supported Providers section below for provider-specific URLs.
Azure API Version	2024-10-21	Required for Azure OpenAI only. The API version to use (e.g. `2024-10-21`). See Azure OpenAI documentation for available versions.
API Key	(optional)	The API key for authenticating with the AI provider. Stored encrypted. Leave empty for providers that do not require authentication (e.g. Ollama, Splunk Hosted).
Model	gpt-4o	The model name to use. Some providers support version aliases (e.g. `gpt-4o`, `mistral-large-latest`) that always target the latest release. See Model Aliases below.
Max Tokens	4096	Maximum number of tokens in the AI response. Higher values allow longer responses but increase cost and latency.
Temperature	0.3	Controls randomness in AI responses. Lower values (e.g. 0.1) produce more deterministic outputs, higher values (e.g. 0.8) produce more creative outputs.
Request Timeout	600	Maximum time in seconds to wait for an AI provider response. The default of 600 seconds (10 minutes) accommodates slow models or initial model downloads (e.g. Ollama pulling a model on first use).
Context Window (Ollama)	8192	Context window size in tokens for Ollama models. Must be large enough to accommodate the system prompt, context data, and conversation history. Only applies to the Ollama provider. Increase this value if context is being truncated.
Custom Prompt	(empty)	Optional additional instructions appended to the system prompt. Use this to customise the AI assistant’s behaviour, add domain-specific context, or enforce response formatting preferences.

Supported Providers¶

Named Providers¶

The following providers are explicitly supported with dedicated dropdown options. All use the OpenAI chat completions API format except Anthropic (which uses its native Messages API) and Splunk Hosted (which uses the SLIM API with automatic SCS authentication).

OpenAI¶

Provider: OpenAI
Base URL: https://api.openai.com/v1
API Key: Required (from platform.openai.com)
Example models: gpt-4o, gpt-4o-mini, gpt-4.1, o4-mini

Azure OpenAI¶

Provider: Azure OpenAI
Base URL: https://<resource-name>.openai.azure.com/openai/deployments/<deployment-name>
API Key: Required (Azure portal)
Azure API Version: Required (e.g. 2024-10-21)
Example models: Use the deployment name configured in Azure

Note

Azure OpenAI uses a different authentication mechanism (api-key header instead of Authorization: Bearer) and requires a mandatory api-version query parameter. TrackMe handles both automatically when the Azure provider type is selected.

Anthropic¶

Provider: Anthropic
Base URL: https://api.anthropic.com/v1
API Key: Required (from console.anthropic.com)
Example models: claude-sonnet-4-5-20250929, claude-sonnet-4-20250514, claude-haiku-4-5-20251022, claude-opus-4-6-20260213

Note

Anthropic does not support -latest aliases. Use either generation-specific aliases (e.g. claude-sonnet-4-5) which auto-update within a generation, or pinned snapshot IDs (e.g. claude-sonnet-4-5-20250929) for deterministic behaviour. See Anthropic’s model documentation for the full list.

Note

Anthropic uses a native Messages API rather than the OpenAI chat completions format. TrackMe automatically handles the API differences when the Anthropic provider type is selected.

Google Gemini¶

Provider: Google Gemini
Base URL: https://generativelanguage.googleapis.com/v1beta/openai
API Key: Required (from Google AI Studio)
Example models: gemini-3-flash-preview, gemini-3-pro-preview, gemini-2.5-flash, gemini-2.5-pro

Note

Google Gemini uses its OpenAI-compatible endpoint for integration. The API key is passed as a Bearer token in the Authorization header, which TrackMe handles automatically. Gemini 3 models are currently in preview and use the -preview suffix in their model names.

Mistral AI¶

Provider: Mistral AI
Base URL: https://api.mistral.ai/v1
API Key: Required (from console.mistral.ai)
Example models: mistral-small-latest, mistral-medium-latest, mistral-large-latest

Ollama¶

Provider: Ollama
Base URL: http://<ollama-host>:11434/v1
API Key: Not required
Example models: mistral, llama3.2, phi3, qwen2.5

Hint

Ollama is a self-hosted inference server that provides full data privacy and control over your AI infrastructure. It runs powerful open-source models without sending any data to external APIs, making it ideal for environments with strict data governance requirements or air-gapped networks. On first use with a new model, TrackMe automatically triggers the model download on the Ollama server, which can take several minutes. The default request timeout of 600 seconds accommodates this.

Splunk Hosted (SLIM API)¶

Provider: Splunk Hosted (SLIM API)
Base URL: Leave empty (auto-discovered)
API Key: Not required (auto-managed)
Example models: gpt-oss-120b, gpt-oss-20b, foundation-sec-1.1-8b-instruct

Important

The Splunk Hosted (SLIM API) provider is only available on Splunk Cloud instances where Splunk has enabled the SLIM (Splunk Language and Intelligence Model) API. It is not available on Splunk Enterprise (on-premises) deployments.

How it works:

Unlike all other provider types, Splunk Hosted requires no manual credential or endpoint configuration:

No Base URL: The SLIM API endpoint is automatically discovered from the Splunk Cloud platform using the SCS (Splunk Cloud Services) tenant information. Leave the Base URL field empty.
No API Key: Authentication uses short-lived SCS tokens that are transparently obtained and refreshed by TrackMe at each request. No API key needs to be entered or managed.
Model selection: You only need to choose a model. Available models can be discovered via the TrackMe REST API (see below).

Configuration steps:

Navigate to Configuration > AI Provider and create a new provider entry
Select Splunk Hosted (SLIM API) as the provider type
Leave Base URL and API Key empty
Enter the desired model name in the Model field (e.g. gpt-oss-120b)
Adjust Max Tokens, Temperature, and other settings as needed
Save the configuration

Discovering available models:

You can query the available SLIM API models using the TrackMe REST API:

| trackme url="/services/trackme/v2/ai/admin/models" mode="get"

This returns the list of models currently available on your Splunk Cloud instance, including their model_name (display name) and model_id (the value to enter in the Model configuration field).

Hint

About the models — gpt-oss-120b is a ~117B parameter Mixture of Experts model suitable for general-purpose tasks requiring high quality. gpt-oss-20b is a smaller ~21B parameter model offering faster responses at lower cost. foundation-sec-1.1-8b-instruct is a security-focused model designed for cybersecurity analysis and security operations use cases. Model availability may vary depending on your Splunk Cloud deployment and entitlements.

Custom (OpenAI-Compatible)¶

The Custom provider type works with any API endpoint that implements the OpenAI chat completions format (POST /chat/completions). This includes a wide range of enterprise and open-source providers:

Provider	Base URL	Example Models
xAI (Grok)	`https://api.x.ai/v1`	`grok-3`, `grok-3-mini`
Together AI	`https://api.together.xyz/v1`	`meta-llama/Llama-3-70b-chat-hf`
Groq	`https://api.groq.com/openai/v1`	`llama-3.3-70b-versatile`, `mixtral-8x7b-32768`
Perplexity	`https://api.perplexity.ai`	`sonar-pro`, `sonar`
Cohere	`https://api.cohere.com/compatibility/v1`	`command-r-plus`, `command-r`
AWS Bedrock	`https://bedrock-runtime.<region>.amazonaws.com/openai/v1`	`anthropic.claude-sonnet-4-20250514-v1:0`
DeepSeek	`https://api.deepseek.com/v1`	`deepseek-chat`, `deepseek-reasoner`
OpenRouter	`https://openrouter.ai/api/v1`	Any model available on OpenRouter
NVIDIA NIM	`http://<nim-host>:8000/v1`	Depends on deployed model
LM Studio	`http://<lmstudio-host>:1234/v1`	Depends on loaded model

Note

When using the Custom provider type, ensure that the API endpoint is reachable from the Splunk search head running TrackMe and that it supports the standard OpenAI chat completions request and response format.

Note

A note on GitHub Copilot — GitHub Copilot is not a supported AI provider. Although community projects exist that proxy Copilot’s internal endpoint (api.githubcopilot.com) as an OpenAI-compatible API, this endpoint is not an officially supported public API. It is designed exclusively for GitHub’s own IDE extensions (VS Code, JetBrains, CLI), and using it through third-party applications violates GitHub’s Terms of Service. Relying on it would expose your organisation to account suspension risk and potential breakage without notice. If you are in the Microsoft/GitHub ecosystem, use Azure OpenAI or OpenAI directly instead — these provide official, stable API access to the same underlying models that power Copilot.

Model Aliases¶

Some providers support version aliases that always point to the latest release of a model family. Using aliases avoids having to update the configuration when a new model version is released:

Mistral: mistral-small-latest, mistral-medium-latest, mistral-large-latest — auto-update to the newest version within each model tier.
OpenAI: gpt-4o, gpt-4o-mini — effectively aliases that point to the latest patch.
Anthropic: Does not support -latest aliases. Use generation-specific aliases (e.g. claude-sonnet-4-5) or pinned snapshot IDs (e.g. claude-sonnet-4-5-20250929).

Check your provider’s documentation for available aliases and version pinning options.

Multi-Provider Setup¶

TrackMe supports configuring multiple AI providers simultaneously. This is useful for:

Model comparison: Try different models on the same question to compare analysis quality
Cost optimisation: Use a cheaper model for routine questions and a premium model for complex investigations
Redundancy: Fall back to an alternative provider if the primary is unavailable
Data privacy: Run Ollama alongside a cloud provider — use the self-hosted option for sensitive environments and the cloud provider for general use
Platform-native option: On Splunk Cloud, use Splunk Hosted alongside external providers — keep data within the Splunk platform while having an external model available as an alternative

To set up multiple providers, create multiple entries in the AI Provider configuration tab, each with a descriptive name (e.g. openai_gpt4o, ollama_mistral, anthropic_claude).

When multiple providers are available, users see a provider selector dropdown in the AI chat panel that displays each provider’s name and model.

Usage¶

Opening the AI Assistant¶

The AI Assistant is available throughout TrackMe via two patterns depending on the context.

From Entity Investigation¶

When investigating a specific entity (DSM, DHM, MHM, FLX, FQM, WLK), the AI Assistant can be opened via:

Entity overview panel:

Open an entity investigation panel by clicking on an entity in the monitoring dashboard
Click the AI Assistant button in the panel header
The AI chat panel opens as a full-screen slide panel with the entity context pre-loaded

Entity actions menu:

Click the three-dot actions menu (kebab menu) on any entity row in the monitoring dashboard
Select Ask AI
The AI chat panel opens directly with the entity context pre-loaded

From TrackMe UI Pages¶

Each TrackMe UI page features an AI assistant button in the top-right header. Clicking it opens the AI chat panel with the relevant context for that page automatically loaded.

The AI Assistant is available from the following pages:

Virtual Tenants — context includes all tenants, their health states, and configuration
Tenant Home — context includes tenant-specific entities and component overview
REST API Reference — context includes the full API catalogue, resource groups, and usage patterns
Backup & Restore — context includes backup records, status, and scheduling information
Global Maintenance Mode — context includes current maintenance status, scheduling, and countdown
Maintenance Knowledge Database — context includes KDB records and per-entity maintenance windows
Bank Holidays Management — context includes holiday periods, country coverage, and recurring calendars
License Management — context includes edition, validity, expiration, and feature information

Note

The AI assistant button is only visible when the feature is enabled in the General settings and at least one AI provider is configured.

Chat Interface¶

The AI chat panel provides a conversational interface with the following features:

Suggested questions: When the chat panel opens, context-specific suggested questions are displayed to help get started. The questions are tailored to the page the assistant was opened from. For example, entity investigation suggests questions about health state and thresholds, while the Backup & Restore page suggests questions about backup status and restore procedures.

Free-form questions: Type any question in the input field at the bottom of the panel. Press Enter to send (use Shift+Enter for a newline).

Streaming responses: AI responses stream in real-time as tokens are generated, providing immediate feedback.

Conversation context: The AI Assistant maintains conversation history within the session. Follow-up questions are understood in the context of previous messages.

Markdown rendering: AI responses support full Markdown formatting including code blocks, tables, lists, and inline formatting. SPL searches are displayed in syntax-highlighted code blocks for easy copy-paste.

Provider switching: If multiple providers are configured, use the dropdown at the top of the chat panel to switch between providers. Switching providers clears the conversation history.

Clear chat: Use the trash icon button to clear the conversation and start fresh.

Context Enrichment¶

When a chat session starts, TrackMe automatically collects structured data relevant to the page the assistant was opened from. This context is passed to the LLM as part of a specialised system prompt, enabling the AI to provide specific, data-driven answers without the user having to describe the situation manually.

The following table summarises what context is collected for each area:

Context	Data Collected
Entity investigation	Health state, scoring, data lag and event count metrics, configured thresholds, ML Outliers detection results, investigation SPL searches, health history, entity metadata (index, sourcetype, host, etc.), and threshold tuning guidance.
Virtual Tenants	All tenants with their health states, component counts, configuration, and license information.
Tenant Home	Tenant-specific entities, health overview, components, and configuration details.
REST API Reference	Full API catalogue including resource groups, endpoints, HTTP methods, authentication patterns, SPL `\| trackme` command syntax, and the `describe=true` self-documentation pattern.
Backup & Restore	Backup archive records (timestamps, labels, status, type), restore history, and a knowledge reference covering backup procedures and best practices.
Global Maintenance Mode	Current maintenance status (enabled/disabled/scheduled), start and end times, countdown to auto-disable, and a knowledge reference on maintenance impact.
Maintenance Knowledge Database	KDB record counts, recent records sorted by end time (planned and unplanned types), per-entity maintenance windows, and guidance on KDB management.
Bank Holidays Management	Holiday periods, country coverage, recurring calendar configuration, and information on how holidays affect SLA calculations and monitoring thresholds.
License Management	Current edition, validity and expiration status, feature availability per edition, developer mode status, and renewal workflow guidance.

Entity Threshold Tuning Guidance¶

The entity investigation context includes a thresholds_help section that provides entity-type-specific advisory guidance for threshold tuning. This enables the AI Assistant to give accurate, actionable advice when users ask about adjusting thresholds:

DSM / DHM: Explains event delay, ingestion latency, future tolerance, adaptive delay, variable delay, and lagging class thresholds. DSM entities additionally include host count monitoring guidance.
MHM: Covers metric lag and future tolerance thresholds, as well as the metric category policy.
FLX: Distinguishes between entity-level settings (inactivity timeout) and use case-level threshold conditions defined in the FLX tracker configuration.
FQM: Explains that thresholds are managed at the FQM tracker level (success rate and coverage conditions).
WLK: Explicitly notes that Workload Monitoring entities do not have user-configurable thresholds, and advises investigating execution issues instead.

Session Lifecycle¶

Panel close and job cancellation: When the AI chat panel is closed (or the user navigates away), the frontend automatically cancels any in-flight AI request. This releases the concurrency slot immediately so other users can use the AI Assistant without hitting the concurrency limit. The backend worker thread may continue briefly until the LLM responds, but the response is discarded.

Custom Prompt¶

Administrators can use the Custom Prompt field in the AI Provider configuration to add domain-specific instructions that are appended to the system prompt. Example use cases:

Enforce response formatting (e.g. always include a “Next Steps” section)
Add organisation-specific context (e.g. known maintenance windows, team ownership)
Customise the assistant’s tone or verbosity
Include relevant runbook references or escalation procedures

AI Status Report in Stateful Alert Emails¶

In addition to the interactive chat assistant, the AI infrastructure powers an optional AI Status Report feature in stateful alert email notifications. When enabled, each email notification includes an AI-generated entity status summary embedded directly in the HTML email body.

How it works:

When a stateful alert triggers and email delivery is required, TrackMe calls the configured AI provider with the entity context (the same describe data used by the interactive chat).
The AI generates a concise, actionable summary tailored to the incident lifecycle stage:
- Opened: Explains what went wrong and where to investigate first
- Updated: Analyses whether the situation has changed since the last notification
- Closed: Confirms the recovery and summarises what was previously wrong
The AI response is converted to email-safe HTML and embedded in the notification between the detailed information and charts sections.

Key design principles:

Same provider configuration: The AI Status Report uses the same AI provider(s) configured in Configuration > AI Provider. No additional AI setup is required.
Fail-open: If the AI provider is unavailable, slow, or returns an error, the email notification is delivered normally without the AI section. No email is ever blocked by an AI failure.
Cost-aware: The AI report is only generated when email delivery is actually needed. If the entity’s priority level is filtered out from email notifications by priority routing, no AI call is made.
Timeout capped: The AI request timeout is capped at 120 seconds for email flows to avoid excessive notification delays.

Example of an email notification with the AI Status Report:

Configuration:

The AI Status Report is configured per stateful alert in the alert creation wizard or in the Splunk alert action editor:

AI Status Report: Enable or disable. Defaults to enabled when at least one AI provider is configured.
AI Provider: Optionally select a specific provider when multiple are configured. Defaults to the first available provider.

For detailed setup instructions, see Alerting Architecture & Third-Party Integration.

Troubleshooting:

AI Status Report operations are logged with the task=ai_status_report tag in the alert action logs:

index=cim_modactions sourcetype=modular_alerts:trackme_stateful_alert "task=ai_status_report"

Error Handling¶

The AI Assistant handles errors gracefully with clear user-facing messages:

Error Type	Description
AI not configured	No AI provider has been configured. An administrator must set up at least one provider in the Configuration page.
AI disabled	The AI assistant has been disabled by an administrator in the General settings.
AI busy	The maximum number of concurrent AI chat requests has been reached. The user should try again in a moment.
API key missing	The AI provider requires an API key but none is configured. This can occur if a provider was created without saving the API key, or if the credential was lost. Re-enter the API key in the AI Provider configuration.
Provider error	The AI provider returned an error (e.g. invalid API key, model not found, rate limit exceeded). The error message from the provider is displayed.

Troubleshooting¶

Logs¶

The AI Assistant logs all requests, errors, and streaming activity to dedicated log files on the Splunk search head. Use the following SPL search to access AI-related logs:

index=_internal sourcetype="trackme:rest_api" source="/opt/splunk/var/log/splunk/trackme_rest_api_ai_*"

Key log events to look for:

Log Pattern	Description
`action="request_started"`	A new AI chat request was submitted. Includes the provider, model, base URL, timeout, context type, and whether context was loaded.
`action="first_token"`	The first token was received from the LLM provider. The `time_to_first_token_sec` field indicates provider responsiveness.
`action="streaming_progress"`	Periodic heartbeat during long-running streaming responses (every 30 seconds). Shows elapsed time and characters received.
`action="timeout"`	The LLM request timed out. Check the `timeout_sec` and `elapsed_sec` fields. Consider increasing the Request Timeout setting.
`action="stale_job_cleanup"`	A running job was detected as stale (exceeded its timeout plus a grace period). The concurrency slot was released automatically.
`action="slot_released"`	A concurrency slot was released, either after job completion or via panel close cancellation.
`exception=`	An error occurred during the AI request. The exception message typically includes the HTTP status code and error body from the provider.

Common Issues¶

“Missing Authorization header” from Google Gemini

The API key is not configured or was not saved correctly for this provider. Edit the provider in Configuration > AI Provider, re-enter the API key, and save.

“Model not found” (HTTP 404)

The model name is incorrect. Verify the exact model ID in your provider’s documentation. Common mistakes include missing version suffixes (e.g. gemini-3-pro vs gemini-3-pro-preview) or using aliases that don’t exist for the provider.

“AI assistant is at maximum capacity”

All concurrency slots are in use. Either wait for an in-flight request to complete, or increase the Max concurrent AI chats setting in Configuration > General. If slots appear stuck, check for stale jobs in the logs — they are automatically cleaned up after the configured timeout plus a grace period.

Streaming responses appear slow or stalled

For cloud providers, this is typically caused by network latency or provider-side rate limiting. For Ollama, ensure the server has sufficient resources (CPU/GPU/RAM) for the model. Check the time_to_first_token_sec in the logs to distinguish between slow initial response and slow token generation.

Context not loaded

If the AI responds with generic advice rather than specific analysis, check the entity_context_loaded field in the request log. A value of False indicates the context collection failed. For entity investigations, verify that the entity exists in the specified tenant and that the user has RBAC permissions to access it. For other contexts, check that the relevant TrackMe backend services are running correctly.