AI Compliance, Privacy and Data Governance¶

Overview¶

This page is written for compliance, security, privacy and risk teams evaluating the use of TrackMe’s AI capabilities in regulated or data-sensitive environments. It answers, in one place, the questions that matter for an AI risk assessment:

Where does the AI processing happen, and who is in the data path?
Exactly what data is exchanged with the LLM provider — and what is never sent?
What can the AI actually do — which tools does it have, and which are read-only?
How are write actions controlled, audited and reversed?
How do we restrict, monitor, or completely disable the AI layer?

Every statement on this page reflects how the product is implemented. For the functional walkthroughs of each AI capability, see Artificial Intelligence in TrackMe and the dedicated pages of this section; for provider configuration, see Configuring AI Provider accounts.

Important

The short version — TrackMe’s AI layer is opt-in and inert by default: no LLM call is ever made until an administrator explicitly configures an AI provider account. When enabled, LLM calls go directly from your Splunk search head to the provider you chose — including fully air-gapped self-hosted models — using your own credentials. TrackMe Limited (the vendor) operates no AI cloud service, no proxy, and no telemetry collection in the AI data path and never receives your data. The AI works on monitoring metadata and telemetry, not on your raw indexed events, and every interaction is recorded in an audit trail you control.

Compliance at a glance¶

Question	Answer
Is AI enabled by default?	No. The AI layer is completely inert until an administrator creates at least one AI provider account. Until then, no data leaves Splunk and no LLM call is made.
Is the vendor (TrackMe Limited) in the data path?	No. TrackMe is a Splunk application running entirely inside your Splunk deployment. LLM requests are sent directly from your search head to the provider endpoint you configured. There is no intermediate TrackMe-operated service, and TrackMe Limited never receives any of your data, prompts, or responses.
Can we keep all AI processing on-premises?	Yes. TrackMe supports fully self-hosted inference via Ollama and any private OpenAI-compatible endpoint (e.g. NVIDIA NIM, LM Studio, corporate LLM gateways) — zero external API calls. On Splunk Cloud, the Splunk Hosted (SLIM API) provider keeps processing inside the Splunk platform. See Configuring AI Provider accounts.
Is our data used to train models?	TrackMe sends requests to API endpoints using your own API keys, under the data-use terms of your agreement with the provider. Major providers (OpenAI, Anthropic, Azure OpenAI, Google, Mistral) contractually do not train on API traffic by default. TrackMe itself never stores or reuses your prompts for any training purpose. See Provider choice, data residency and model training.
Are raw Splunk events sent to the LLM?	No, with one narrow, documented exception. AI context is built from TrackMe’s own monitoring metadata and telemetry (entity state, thresholds, metric histories, audit records). The only feature that can expose event samples is the FQM Advisor’s sampled-failure inspection, scoped to field-quality diagnosis. See What is sent to the LLM — and what is not.
Are credentials or secrets ever sent to the LLM?	No. API keys, Splunk credentials, session tokens and stored passwords are never part of any prompt or context payload. Provider API keys are stored encrypted in Splunk’s `storage_passwords` and are write-only once saved.
Can the LLM run arbitrary searches in our environment?	No. The LLM never generates raw SPL for execution. All data access goes through a fixed registry of typed tools whose SPL and REST calls are pre-built and parameter-validated by TrackMe. See The complete tool inventory.
Can the AI change things without a human?	Not unless you explicitly opt in. Interactive advisor/Concierge writes always require an explicit user consent click (executed under the user’s own credentials and RBAC). Unattended advisor automation is disabled by default, per-tenant, and protected by additional guards (e.g. the agent can never decommission an entity or disable a model unless you enable a dedicated flag). AI Routines — user-authored scheduled tasks — are notify-only unless the author selects act mode and grants explicit capability-scoped permissions under a hash-pinned consent captured at creation; editing the intent or the grants invalidates the consent and the routine cannot act again until re-affirmed. See AI Routines: user-authored automation, explicitly consented.
Is AI activity audited?	Yes, comprehensively. Every chat, every advisor run, and every individual tool call is indexed to your Splunk environment with user, provider, model, token usage, duration and outcome. Every write is additionally stamped in the per-entity audit trail with an `[AI Agent] <reason>` justification. See Auditing AI activity.
Can we restrict who uses AI?	Yes. AI endpoints are gated by TrackMe’s standard role model (read / power / admin capabilities, per-tenant role scoping), enforced at the REST boundary. See Access control (RBAC).
Who controls retention of AI audit data?	You do. AI audit events are indexed to your tenant summary index and follow your Splunk index retention policies; no AI data is retained outside your Splunk deployment by TrackMe.

Architecture and data flow¶

Where the AI runs¶

TrackMe’s AI layer runs entirely inside your Splunk deployment:

A user (or a scheduled automation you enabled) triggers an AI interaction in TrackMe.
The TrackMe backend, running on your search head, builds the request: a system prompt, the relevant monitoring context, and — for agentic features — the typed tool registry.
The search head makes a direct HTTPS call to the AI provider endpoint configured in Configuring AI Provider accounts, authenticated with your API key (or no key at all for self-hosted Ollama and Splunk Hosted).
The response is rendered to the user, and the interaction is audited to your tenant summary index.

There is no third-party relay: the only parties in the data path are your Splunk platform and the AI provider you selected. If you select a self-hosted provider, no data leaves your network at all.

Note

TrackMe Limited never receives your data. The vendor operates no AI backend, collects no AI telemetry, and has no technical ability to observe your prompts, your monitoring data, or the provider responses. The AI provider relationship — endpoint, credentials, region, data-use terms — is entirely between your organisation and the provider you configure.

Agentic execution model¶

The conversational AI Assistant is a single-call interface: TrackMe builds a context payload (see below), sends one request, and renders the answer. The assistant executes no tools and performs no writes — when an action would help, it can only propose one via a consent card that the user must explicitly approve.

The AI Advisors and the AI Concierge are agentic: they run a bounded reasoning loop (built on the Splunk Agent SDK, splunklib.ai) in which the LLM may call tools from a fixed, tag-filtered registry. The tools — not the LLM — perform all data access, through TrackMe’s own REST API and pre-built searches, under Splunk authentication. Every tool invocation is individually audited.

The AI Routines are agentic on the same SDK and tool registry, but are user-authored and scheduled rather than product-defined: an operator describes a monitoring task in plain English and TrackMe runs it on a cron schedule. They are the one surface where a write can occur on a schedule without a per-action click — so their governance is described in full below.

AI Routines: user-authored automation, explicitly consented¶

AI Routines let an operator schedule a natural-language task that watches a condition and then notifies or acts. Because a routine can act autonomously on its schedule, it is governed by a distinct, stricter consent model than the interactive advisors:

Notify by default, act only on opt-in. A routine in notify mode has no write tools — it can only deliver a message. Writing requires the author to switch it to act mode and grant specific permissions; a notify routine can never change configuration.
Capability-scoped permission families. Act permissions are granted as 16 capability-oriented families (e.g. entity thresholds, priority, labels/notes, ML models, alert actions). Ticking a family exposes exactly its tools at fire time and nothing more — there are no implied grants.
Hash-pinned, perishable consent. At creation the author affirms an explicit consent that is cryptographically pinned to the verbatim intent, the exact set of granted families, and — where applicable — the exact alert-action contract and the external-MCP tool-name list. Any edit to those invalidates the consent, and the routine cannot act again until the author re-affirms. The consent is checked on every fire.
Deterministic, non-LLM delivery. Recipients (email) and event targets (index/sourcetype) are configuration, never model output — the LLM authors the content, never the audience. Alerts and SPL built on routine events cannot be redirected by a model phrasing something differently.
Off-box side effects are gated. Two families reach outside TrackMe’s own KV Store — trigger Splunk alert actions (open incidents / notify ITSM, Slack, PagerDuty) and execute external MCP tools (the Splunk MCP Server). Both are act-only, part of the pinned consent, and enforced at the Splunk boundary: the MCP token is minted for the routine’s owner under Splunk RBAC, exposing only the explicitly allow-listed tool names.
Identities and tenant scope are never model-authored. A routine runs as a resolved owner (the tenant’s service account, or the creator) recorded on every event; the tenant it may touch is fixed from the routine’s own scope, not the LLM — a tool call targeting any other tenant is refused before it executes.
Who may create routines is policy-controlled. A deployment-wide creator-roles setting restricts routine creation to administrators, or to administrators and power users; global (cross-tenant) routines always require an administrator. The whole feature has its own kill switch (enable_ai_routines) and is inert until the AI layer is enabled and a provider configured.

Every routine fire — reads, writes and the fire/no-fire decision — is audited exactly like an advisor run (see Auditing AI activity), with act-mode writes additionally stamped [AI Agent] <reason> in the per-entity trail.

What is sent to the LLM — and what is not¶

Data sent, per feature¶

All AI context is built server-side by TrackMe from its own collections and summary telemetry. The table below describes the maximum content each feature can include in a request to the LLM:

Feature	Data included in LLM requests
AI Assistant (chat)	The TrackMe system prompt for the current page context; the user’s typed message and the conversation history of the current chat; and a structured context payload built from TrackMe’s describe endpoints — entity identity and configuration (names, identifiers, state, thresholds, schedules), monitoring telemetry (lag/latency/volume KPIs, outlier scores), active alerts and Guardian configuration findings, tenant/component summaries, and license state on the relevant pages. Context is rebuilt fresh on every request.
AI status reports (stateful alert emails)	The same class of entity context as the AI Assistant, for the alerting entity, used to generate the short AI investigation summary embedded in the alert email.
AI Advisors (ML, Feed Lifecycle, FLX Threshold, FQM, Component Health)	Entity configuration and state; metric and quality histories (e.g. 7–30 day delay, latency, outlier-score or quality-percentage series, aggregated via `mstats`/summary searches); peer-entity threshold comparisons; ML model training metadata (distributions, boundaries, confidence); per-entity audit/state-transition history; and the optional free-text instruction the operator provides when launching the run.
AI Concierge	The TrackMe REST API catalog metadata (endpoint paths, schemas, descriptions, danger levels — product documentation, not customer data); the user’s request; and the responses of read-only catalog-validated endpoint calls the agent performs to verify state.
AI Agents automation (scheduled)	The same per-entity context as the corresponding advisor, for the entities selected by your per-tenant scope filters (priority filter and optional filter expression).
AI Routines	The operator’s plain-English intent and its compiled instruction; a slimmed per-tenant entity summary (state, priority, recency of state changes) for the routine’s scope; the trigger-match rows for hybrid/deterministic routines; and the routine’s own memory — compact fingerprints of its recent runs and, for entity-level intents, the exact entity names it acted on last (see AI Routines). No credentials, no user directory data, no raw indexed events. Anonymisation toggles apply to routine context as they do elsewhere.

What is never sent¶

The following are structurally excluded from every AI request:

Raw indexed events from your Splunk indexes — with the single exception documented below. TrackMe’s AI reasons over monitoring metadata and aggregated telemetry, not over your log data.
Credentials and secrets — Splunk passwords, session keys, API keys (including the AI provider keys themselves), and tokens are never included in prompts, context payloads, or tool results.
User directory data — usernames and email addresses of your Splunk users are not part of context payloads (the operator’s identity is recorded in the local audit trail, but not sent to the LLM).
Data from other configured providers — provider accounts are isolated; no data is ever shared between providers.

Important

The FQM exception — the AI FQM Advisor includes a get_fqm_sampled_failures tool that lets the agent inspect samples of failing events already collected by the FQM collect job, strictly to diagnose field-quality issues (e.g. test a regex against real values). If your FQM-monitored sourcetypes contain sensitive payloads and your policy prohibits sending event samples to your chosen provider, either run the FQM Advisor against a self-hosted provider, or do not enable the FQM Advisor. All other advisors and the assistant operate on metadata and aggregated telemetry only.

Anonymisation options¶

For organisations whose infrastructure metadata itself is sensitive (hostnames, IPs, index names), TrackMe provides global anonymisation toggles (Configuration > General > Artificial Intelligence, see Configuring AI Provider accounts):

Setting	Effect
Anonymize entity names for AI	Entity names (object and alias fields) are replaced by SHA256 hashes in every payload sent to the AI provider. The AI is guided to reason on opaque `object_id` references instead; when an action is approved, the real identifiers are resolved locally by TrackMe — the LLM never constructs primary keys.
Anonymize index names for AI	Splunk index names are replaced by SHA256 hashes in entity information, investigation searches and metric details sent to the provider.

Both options trade some answer readability for confidentiality and can be combined with a self-hosted provider for defence in depth.

Provider choice, data residency and model training¶

You decide where data is processed¶

TrackMe is provider-agnostic — the same AI code runs against every supported backend, so the choice of jurisdiction, deployment model and data-use terms stays with you:

Requirement	Option
Air-gapped / fully on-premises	Ollama (self-hosted open-source models) or any internal OpenAI-compatible endpoint (NVIDIA NIM, LM Studio, corporate gateways). Zero external calls.
Keep data inside the Splunk platform (Splunk Cloud)	Splunk Hosted (SLIM API) — models operated by Splunk within the Splunk Cloud platform; no external API key, short-lived platform tokens, no third-party egress.
Specific geography / sovereign cloud	Azure OpenAI deployed in your chosen region, or region-pinned deployments from other providers via the Custom provider type.
Managed frontier models	OpenAI, Anthropic (Claude), Google Gemini, Mistral AI, xAI (Grok) — via their public API endpoints with your own account.

See Configuring AI Provider accounts for the full configuration reference of each provider type.

Model training and provider-side retention¶

Three facts frame the training question:

TrackMe never trains on, stores, or reuses your AI traffic. Prompts and responses transit the search head and are rendered to the user; what TrackMe persists is the audit metadata and the structured advisor results, in your Splunk indexes.
All cloud providers are consumed through their API tier, authenticated with your organisation’s API key. The applicable data-use and retention terms are those of your agreement with the provider — not consumer-product terms. As of this writing, the major supported providers (OpenAI, Azure OpenAI, Anthropic, Google, Mistral) state that API/business traffic is not used to train their models by default, and several offer zero-data-retention or regional-processing agreements on request.
Self-hosted options remove the question entirely — with Ollama or a private endpoint, no third party ever sees the data.

Note

Provider terms evolve. For a formal assessment, verify the current data-usage commitments directly with your provider — e.g. OpenAI’s API data usage policies, Anthropic’s commercial terms, Microsoft’s Data, privacy, and security for Azure OpenAI, Google’s Gemini API terms, Mistral’s privacy policy — and attach them to your AI risk file alongside this page.

Credential management¶

AI provider API keys are entered once in Artificial Intelligence > Manage AI Provider accounts and stored encrypted in Splunk’s secure credential store (storage_passwords). They are write-only: the UI and REST API never return a stored key.
Decryption is performed server-side, only at the moment an LLM request is made, by TrackMe’s backend under Splunk’s access controls. Keys are never logged, never included in audit events, and never part of any prompt.
Self-hosted (Ollama, custom internal endpoints) and Splunk Hosted providers require no stored API key at all.

The complete tool inventory¶

The compliance-critical property of TrackMe’s agentic features is that the LLM can only interact with your environment through a fixed registry of typed tools. There is no generic “run a search” or “call an API” capability exposed to the model:

The LLM never writes SPL. Every search a tool performs is pre-built by TrackMe (mstats on TrackMe metric indexes, bounded searches on TrackMe summary/audit indexes); the model only supplies validated, typed parameters.
Tool sets are mode-filtered at runtime. In inspect mode, write tools are absent from the registry — not merely discouraged. Act mode adds the write tools and is gated by RBAC.
Every write tool requires a ``reason`` that is stamped into the per-entity audit trail as [AI Agent] <reason>, and every tool invocation (read or write) emits its own audit event.

The tables below enumerate every tool, per advisor.

AI Assistant (chat)¶

The assistant has no tools and no write path. It receives context built server-side and can only propose advisor or Concierge invocations as consent cards; nothing executes without the user’s explicit click, which runs under the user’s own credentials.

AI Concierge¶

Catalog-driven generalist with six read tools and zero write tools by design — mutations are proposed as action contracts and executed only by the user’s consent click, through the user’s own RBAC-checked credentials. Destructive actions additionally require the user to type the action name to confirm.

Tool	Purpose / data accessed	Access
`discover_endpoints`	Search the TrackMe REST API catalog by intent (endpoint metadata only).	Read
`list_resource_groups`	List the curated resource groups of the API catalog.	Read
`list_endpoints_in_group`	Enumerate the endpoints of one group with their schemas.	Read
`describe_endpoint`	Full self-documentation of a single endpoint (parameters, examples, danger level, RBAC).	Read
`read_via_endpoint`	Execute a read-classified endpoint to verify live state. Refuses any path not present in the catalog or not marked read-only.	Read
`propose_action`	Validate and record a proposed action contract — does not execute. The consent card click is what fires the actual REST call.	Read (proposal only)

AI ML Advisor¶

Scope: ML outlier model integrity. See AI ML Advisor.

Tool	Purpose / data accessed	Access
`get_entity_outlier_context`	Entity description and ML model state (config, boundaries, confidence).	Read
`get_entity_metric_history`	Metric time series from TrackMe metric indexes (`mstats`).	Read
`get_model_training_details`	Fitted distributions, boundaries, fit-quality indicators.	Read
`get_outlier_score_history`	Historical outlier scoring timeline.	Read
`get_entity_alert_history`	Entity state-transition / alert history from the audit trail.	Read
`get_model_render_history`	Training-window rendering with boundaries and outlier flags.	Read
`add_period_exclusion`	Exclude an anomalous window from model training. Reversible.	Write
`trigger_model_retrain`	Retrain the model (live or simulation).	Write
`set_false_positive`	Neutralise the current outlier score as a false positive.	Write
`update_model_rules`	Tune model parameters; refuses empty/no-op updates.	Write
`manage_outlier_detection`	Enable / disable / reset outlier detection. Disabling in automated mode is blocked unless the dedicated tenant flag is set.	Write (guarded)

AI Feeds Advisor (Feed Lifecycle — DSM/DHM)¶

Scope: data source / data host thresholds and lifecycle. See AI Feeds Advisor.

Tool	Purpose / data accessed	Access
`get_entity_lifecycle_context`	Full entity lifecycle context (config, thresholds, state, alerts).	Read
`get_entity_delay_latency_history`	30-day delay/latency history from the summary index.	Read
`get_entity_alert_flip_history`	State-transition (flip) history from the audit index.	Read
`get_entity_peer_comparison`	Threshold comparison against peer entities of the same tenant.	Read
`get_entity_variable_delay_schedule`	Current variable-delay schedule, if any.	Read
`get_entity_adaptive_delay_history`	Adaptive-delay change history from the audit trail.	Read
`update_entity_thresholds`	Update delay / latency thresholds.	Write
`update_entity_monitoring_state`	Enable or disable monitoring of the entity. Disabling in automated mode is blocked unless the decommission flag is set.	Write (guarded)
`update_entity_adaptive_delay`	Lock or unlock adaptive delay.	Write
`update_entity_variable_delay`	Create or update a variable-delay schedule.	Write
`update_entity_priority_and_tags`	Update entity priority and/or manual tags.	Write
`update_entity_impact_score_weights`	Adjust per-entity impact-score weights.	Write

AI FQM Advisor¶

Scope: field-quality thresholds and the data dictionary. See AI FQM Advisor.

Tool	Purpose / data accessed	Access
`get_fqm_entity_context`	Full FQM entity context.	Read
`get_fqm_dictionary`	Data dictionary and the trackers using it.	Read
`get_fqm_field_dictionary_entry`	A single field’s dictionary entry.	Read
`get_fqm_quality_history`	Success / coverage percentage history with day-of-week breakdown.	Read
`get_fqm_sampled_failures`	Sampled failing events collected by the FQM collect job (see the FQM exception above).	Read
`get_fqm_collect_job_context`	Health of the collect-job and monitor saved searches.	Read
`get_fqm_peer_entity_thresholds`	Peer FQM entities’ thresholds on the same tracker.	Read
`get_fqm_datamodel_context`	CIM datamodel context and diff versus expected fields.	Read
`test_fqm_regex`	Test a candidate regex against real sampled values.	Read
`add_fqm_threshold` / `update_fqm_threshold` / `delete_fqm_threshold`	Manage field-quality thresholds.	Write
`update_fqm_field_dictionary_entry` / `update_fqm_dictionary_bulk`	Update data-dictionary validation configuration (single field or bulk).	Write
`update_fqm_entity_state_priority`	Update monitoring state or priority.	Write (guarded)

AI FLX Threshold Advisor¶

Scope: Flex Object dynamic thresholds. See The AI Advisors.

Tool	Purpose / data accessed	Access
`get_flx_entity_context`	Full FLX entity context.	Read
`get_flx_metric_history`	30-day metric value history (`mstats`).	Read
`get_flx_threshold_breach_history`	State-transition and threshold-breach history.	Read
`get_flx_use_case_definition`	The use-case definition of the entity’s tracker.	Read
`get_flx_peer_entity_thresholds`	Peer FLX entities’ thresholds on the same tracker.	Read
`add_flx_threshold` / `update_flx_threshold` / `delete_flx_threshold`	Manage dynamic thresholds.	Write
`set_flx_variable_threshold_slots`	Configure time-slot-based variable threshold values.	Write
`update_flx_entity_state_priority`	Update monitoring state or priority.	Write (guarded)

AI Component Health Advisor (WLK / MHM)¶

Scope: workload (WLK) and metric-host (MHM) health. See AI Component Health Advisor. The tool registry is component-filtered at runtime — a WLK run only sees WLK tools, an MHM run only MHM tools.

Tool	Purpose / data accessed	Access
`get_wlk_entity_context`	Full WLK entity context.	Read
`get_wlk_execution_history`	Execution history with day-of-week breakdown.	Read
`get_wlk_breach_history`	State-transition and threshold-breach history.	Read
`get_wlk_peer_thresholds`	Peer WLK entities in the same Splunk app.	Read
`add_wlk_threshold` / `update_wlk_threshold` / `delete_wlk_threshold`	Manage execution-metric thresholds.	Write
`update_wlk_entity_state_priority`	Update monitoring state or priority.	Write (guarded)
`get_mhm_entity_context`	Full MHM entity context.	Read
`get_mhm_metric_lag_history`	Metric-lag history with day-of-week breakdown.	Read
`get_mhm_breach_history`	State-transition and lag-breach history.	Read
`get_mhm_lagging_classes`	Lagging-class definitions for the tenant.	Read
`update_mhm_metric_max_lag`	Update the metric-lag threshold.	Write
`update_mhm_entity_state_priority`	Update monitoring state or priority.	Write (guarded)

Shared metadata tools (all specialised advisors, act mode only)¶

Tool	Purpose / data accessed	Access
`manage_entity_labels`	Assign or remove labels from the tenant’s curated catalog. Does not auto-create labels.	Write
`manage_entity_note`	Create or delete free-form entity notes (append-only; notes are immutable once created).	Write

AI Routines (act mode)¶

A routine draws from the same fixed, typed tool registry as the advisors — there is no routine-specific “run anything” capability. Read tools (the tenant/scope summary, a bounded read-only correlation search, inspect-only advisor dispatch, and the flip-history reader) are always available; write tools are exposed only for the capability families the author consented to, one coherent capability per family (the 16 families are enumerated in AI Routines). Two families reach off-box under the pinned consent and Splunk RBAC — trigger Splunk alert actions and execute external MCP tools — as described in AI Routines: user-authored automation, explicitly consented. Notify-mode routines receive no write tools at all.

Guardrails and safety controls¶

Defence in depth, by layer:

Layer	Control
Structural	The chat assistant has no tools and no write path. Write tools are physically absent from the registry in inspect mode. The Concierge has zero static write tools. The LLM never constructs primary keys (tenant and entity identifiers come from session state, not model output) and never emits raw SPL.
Human-in-the-loop	Every interactive mutation goes through a consent card: the user reviews the exact proposed action (and can edit the request body for Concierge actions) before clicking. The click executes through the user’s own credentials. Destructive Concierge actions require typed confirmation of the action name.
RBAC	Enforced at the REST boundary, not in the prompt: inspect requires TrackMe read capability, act requires power-level capability, provider administration requires admin capability — all additionally scoped by per-tenant roles. An unauthorized write returns HTTP 403 regardless of what the agent attempts. See Access control (RBAC).
Automation gates	Unattended runs are disabled by default per tenant and per advisor family. Even when enabled, dedicated flags (both disabled by default) are required before an automated agent may decommission an entity (`allow_decommission`) or disable an ML model (`allow_model_disable`). Entity selection is scoped by priority filters and an optional filter expression; an unparseable expression fails closed (no entity processed).
Runtime budgets	Hard token, step and wall-clock budgets per run, enforced by SDK middleware plus an independent watchdog thread; per-search-head concurrency caps; per-tenant Concierge rate limits and per-proposal action caps. A misbehaving run hits a real wall, not a soft prompt.
Validation	Advisor outputs are typed (Pydantic schemas) — invalid completions are retried or rejected. Concierge action paths are validated byte-for-byte against the live REST catalog at proposal time and re-validated at execution time; write tools refuse no-op updates.
Fail-safe semantics	AI failures never block operations: a failed chat degrades to an error message, a failed AI status report is replaced by “AI investigation unavailable” while the alert email still sends, and an unavailable provider simply disables the feature with a clear message.

Access control (RBAC)¶

AI features inherit TrackMe’s standard three-tier role model (see Roles & access control), enforced at every REST endpoint:

Operation	Required capability	Notes
Use the AI Assistant; launch advisors in inspect mode; view audit dashboards	`trackmeuseroperations`	Read-only surface. Audit views and advisor context are additionally filtered to the tenants visible to the caller.
Launch advisors in act mode; approve consent-card actions	`trackmepoweroperations`	Writes execute under the approving user’s credentials and are tenant-scope checked.
Manage AI provider accounts and global AI settings	`trackmeadminoperations`	Provider CRUD, connectivity tests, anonymisation and feature toggles.

Per-tenant role instances (trackme_user|power|admin_<tenant_id>) apply on top of the global capabilities — holding a global capability is not sufficient without the per-tenant role. Scheduled automation runs under the Splunk system user within the per-tenant gates described above.

Audit and traceability¶

Every AI interaction leaves a queryable trace in your Splunk environment — see Auditing AI activity for the dashboards. Three layers of records are emitted:

Record	Content
Per tool call (`trackme:ai_agent:tool_call`)	One event per tool invocation: timestamp, advisor, tool name, status, duration, tenant, component, entity, mode (inspect/act), interactive vs automated, job correlation ID, result size, error details.
Per run (`trackme:ai_agent:<advisor>:<mode>`, `trackme:ai_agent:ai_routine:*`, `trackme:ai_assistant:chat`, `trackme:ai_assistant:status_report`)	One terminal event per advisor run, routine fire or chat: user (or resolved owner) identity, provider and model used, token usage, steps taken, duration, status, how it was launched (UI / chat / automation), the chat session correlation ID, and — for advisors and routines — the full structured result (recommendations and actions taken).
Per entity write (per-entity audit trail)	Every write is stamped `[AI Agent] <reason>` in the affected entity’s Audit changes panel, with the agent required to cite the field changed, the before/after values and the operational trigger.

Properties relevant to compliance review:

Attribution — every event carries the operator’s identity (or automated for scheduled runs), the tenant, the provider account and the exact model. The chat_session_id field lets you answer “which conversation triggered this change?” with one SPL query.
Chat content is not indexed — audit events for chats record metadata and token counts, not the conversation text. Chat exchanges transit a KV Store collection only for asynchronous delivery to the browser and are automatically purged within minutes; they are never written to a Splunk index. Advisor events do persist the structured result, which is the operational record of what was recommended and done.
Retention is yours — all audit events land in your tenant summary index and follow your Splunk retention policies; nothing is retained outside your deployment.
Cost observability — token usage is recorded per run and aggregated in the audit dashboards, per provider and model.

Enablement controls and defaults¶

The complete switchboard, from most global to most granular:

Control	Default	Effect
AI provider account configured	None	The root opt-in. With no provider account, the entire AI layer is inert — no UI surface, no LLM call, no egress.
Enable AI assistant (global, Configuration > General)	Enabled	Master feature switch. When off, every AI endpoint returns HTTP 403 and all AI UI is hidden. Enabled by default for discoverability, but functionally inert until a provider exists.
Anonymize entity / index names for AI (global)	Off	SHA256-hash entity and/or index names in all payloads sent to providers.
Max concurrent AI chats (global)	10	Concurrency cap; excess requests are rejected with HTTP 429.
ML Advisor automation (per tenant)	Disabled	Opt-in scheduled runs; mode, provider, review-frequency and runtime caps per tenant.
Components Advisor automation (per tenant)	Disabled	Opt-in scheduled runs for Feed Lifecycle / FLX / FQM / Component Health; component list, mode and caps per tenant.
Enable AI Routines (global, Configuration > General)	Enabled	Feature switch for AI Routines specifically (on top of Enable AI assistant). When off, routine menus are hidden, the create / evaluate / run-now endpoints return 403, and scheduled routine execution is refused.
Who can create AI Routines (global)	Admins & power users	Restricts routine creation to administrators, or administrators and power users; global (cross-tenant) routines always require an administrator.
Routine act mode + permission families (per routine)	Notify / none	A routine is notify-only unless the author selects act and grants specific capability families, under a hash-pinned consent that any later edit invalidates.
Automation scope filters (per tenant)	`critical,high`	Priority filter plus optional filter expression restricting which entities automation may touch. Invalid expressions fail closed.
Allow decommission / allow model disable (per tenant)	Disabled	Without these flags, automated agents can never disable an entity or an ML model.
Per-entity opt-out	—	Individual entities can be flagged to be skipped by automated advisor runs.
RBAC	—	Standard TrackMe roles gate every AI surface per user and per tenant (see above).

To completely disable AI: do not configure any provider account (or delete existing ones), and set Enable AI assistant to off. Both conditions are independently sufficient to stop all LLM calls.

Network egress and transport security¶

All LLM traffic is outbound HTTPS from the search head(s) to the single endpoint of each configured provider. For firewall/egress allow-listing:

Provider	Egress destination (TCP 443)
OpenAI	`api.openai.com`
Azure OpenAI	`<your-resource>.openai.azure.com` (your region)
Anthropic	`api.anthropic.com`
Google Gemini	`generativelanguage.googleapis.com`
Mistral AI	`api.mistral.ai`
xAI (Grok)	`api.x.ai`
Ollama / Custom (OpenAI-compatible)	Your own endpoint — typically internal; no Internet egress required.
Splunk Hosted (SLIM API)	Splunk Cloud platform-internal (auto-discovered); no customer-managed egress.

Transport properties:

TLS certificate verification uses the system CA trust store and is strict for all cloud providers. For Ollama only, hostname/certificate verification is relaxed exclusively for loopback addresses (127.0.0.1 / localhost); a remote Ollama or custom endpoint is verified strictly.
No inbound connectivity is required by any AI feature; TrackMe never opens listening ports for AI.
Requests are bounded by the per-provider request timeout (see Configuring AI Provider accounts).

Frequently asked compliance questions¶

Does TrackMe Limited (the vendor) see our prompts, data, or AI responses?: No. TrackMe is a Splunk app executing in your environment; there is no vendor-operated AI service, proxy, or telemetry in the data path. The only external party — if you choose a cloud provider — is that provider, under your own account and terms.
Is anything sent to an LLM before we configure a provider?: No. Without a provider account the AI layer is inert: no call, no egress, no background AI processing of any kind.
Which data crosses the boundary to the provider, in one sentence?: The system prompt, the user’s question or operator instruction, and a TrackMe-built context of monitoring metadata and aggregated telemetry for the entity/tenant in scope — never credentials, never user directory data, and never raw indexed events (FQM sampled failures being the single documented, avoidable exception).
Can the AI exfiltrate data by running its own searches?: No. The LLM cannot emit SPL or arbitrary REST calls. It can only invoke the typed tools listed on this page, whose queries are pre-built, scoped to TrackMe’s own collections and indexes, bounded in time range, and individually audited. The Concierge’s only generic read path refuses any endpoint not present in the catalog and not classified read-only.
Can the AI delete data, entities, or tenants on its own?: No. Interactive writes require an explicit user consent click (RBAC-checked, user credentials); destructive Concierge actions additionally require typed confirmation. Automated runs are opt-in, disabled by default, and structurally prevented from decommissioning entities or disabling models unless you enable dedicated flags that are also disabled by default.
Under whose identity do AI actions run?: Interactive reads/writes execute under the launching/approving user’s Splunk session and roles. Scheduled advisor automation runs under the Splunk system user, inside the per-tenant gates and filters you configured. An AI Routine runs as a resolved owner — the tenant’s service account, or the routine’s creator — recorded on every event. All paths are fully attributed in the audit trail.
Can an AI Routine act on its own, and how is that controlled?: Only if its author explicitly made it an act routine and granted specific capability families under a hash-pinned consent. A routine is notify-only by default; in act mode it may apply just the consented capabilities on its schedule, with no per-fire click — but any edit to the intent or the grants invalidates the consent and stops it acting until re-affirmed, delivery targets are fixed configuration (not model output), the tenant it can touch is fixed from its scope, and every fire (including each write, stamped [AI Agent] <reason>) is audited. The whole feature has a global kill switch and a policy for who may create routines. See AI Routines: user-authored automation, explicitly consented.
Is the conversation history stored?: Chat conversations are ephemeral. Message exchanges pass through a KV Store collection solely for asynchronous delivery and are automatically purged within minutes; chat bodies are never indexed. Audit events record metadata only (user, model, tokens, duration, status). Advisor runs persist their structured results as the operational record.
How would we investigate an AI-made change six months later?: The per-entity Audit changes panel shows the [AI Agent] <reason> entry; the summary index holds the per-run and per-tool events with the user, provider, model, and chat_session_id correlation back to the conversation that proposed it — all queryable in plain SPL, retained per your index policies.
Can we run a different provider per sensitivity tier?: Yes. Multiple provider accounts can coexist (e.g. self-hosted Ollama for sensitive tenants, a cloud provider elsewhere), and the provider is selected per chat / per advisor run. No data is shared between providers.
Does using TrackMe AI change our Splunk data’s exposure to Splunk (the company)?: No, with one explicit exception you control: selecting the Splunk Hosted (SLIM API) provider on Splunk Cloud sends AI requests to models operated by Splunk inside the Splunk Cloud platform. All other providers bypass Splunk entirely; self-hosted providers bypass everyone.
What should our DPIA / AI risk assessment reference?: This page (data flows, tool inventory, controls), Configuring AI Provider accounts (provider terms, residency), Auditing AI activity (evidence and monitoring), your chosen provider’s API data-use terms, and your Splunk index retention configuration for the audit trail.