Auditing AI activity¶
Overview¶
Auditability and traceability are first-class principles of the TrackMe AI layer. Every AI Assistant chat, every AI Advisor run, and every Concierge action is indexed to the tenant summary index and surfaced in dedicated audit dashboards — so you can always answer who launched what, which provider and model were used, what was proposed or changed, and how long it took.
There are four audit dashboards under the Artificial Intelligence menu:
Review AI Advisor activity — interactive (UI- or chat-launched) advisor runs.
Review Scheduled AI Advisor activity — the automated nightly batch (covered in AI Agents automation).
Review AI Routines activity — user-authored scheduled routines, interactive and scheduled.
Review AI Assistant activity — chat conversations and stateful-alert AI status reports.
Note
In addition to the dashboards, every advisor or Concierge write is stamped with an [AI
Agent] <reason> prefix in the affected entity’s Audit changes panel, so the per-entity history
shows exactly what the AI changed and why — independently of the dashboards.
Review AI Advisor activity¶
This dashboard gives end-to-end visibility into AI Advisor activity across every tenant — who launched what, which providers and tools were used, and how long each call took. It covers the interactive path; scheduled runs live in the Review Scheduled AI Advisor activity dashboard.
The KPI strip shows total runs, success / error rate, total tokens, average run duration, and active tenants. Donut charts break activity down by advisor, by status, by mode (inspect / act / dictionary_generate), and by top provider-model. A tool-call leaderboard and a recent-runs table complete the picture.
The MCP tool-call leaderboard shows every tool the advisors invoked, with call counts, average and
p95 durations, and success / error counts — useful for spotting a slow or failing tool. The recent
advisor runs table lists each run with its advisor, mode, status, tenant, component, object, user
(or automated), duration, tokens, and the number of recommendations and actions taken.
Tip
Every panel has a 3-dot menu with Open in Search that deep-links into the Splunk Search app
with the panel’s SPL pre-loaded. The underlying events use the sourcetype pattern
trackme:ai_agent:*advisor* (plus trackme:ai_agent:tool_call for per-tool events), so you
can pivot AI activity in plain SPL.
Review AI Routines activity¶
This dashboard gives end-to-end visibility into AI Routines — the user-authored, scheduled natural-language tasks that watch a condition and then notify or act. It combines interactive runs (Evaluate dry-runs and Run-now) with the scheduled fires, so you can see how often each routine ran, whether it fired, what it cost, and how it was launched.
The KPI strip shows total runs, success / error rate, total tokens, average run duration, and active tenants. Donut charts break activity down by routine, by status (fired / no-fire / error), by top tenants, and by top provider-model. Runs-over-time column charts (by routine and by status) and an hour-of-day cadence histogram make it easy to confirm a routine’s schedule is actually firing in its window, and a per-tenant × routine last-run matrix highlights any routine that has gone silent.
Each routine fire is indexed to the tenant summary index with the sourcetype pattern
trackme:ai_agent:ai_routine:* (carrying advisor="ai_routine"), alongside the same per-tool
trackme:ai_agent:tool_call events the advisors emit — so the routine’s reads and writes are
auditable in plain SPL, and each act-mode write is additionally stamped [AI Agent] <reason> in
the affected entity’s Audit changes panel. See AI Routines for the routine model itself and
its per-routine execution logs.
Review AI Assistant activity¶
This dashboard covers the conversational AI Assistant — both interactive chats and the AI status reports embedded in stateful-alert emails. It records who launched each chat, which provider and model answered, the context it ran in, and how long it took.
The KPI strip shows total runs, success / error rate, skipped status reports, total tokens, average duration, and active tenants. Donut charts break activity down by kind (interactive chat vs status report), by status, by context (entity / Tenant Home / wizard / feature pages), and by top provider-model. A status-report skip-reasons panel explains any skipped email reports.
The recent runs table lists each chat or status report with its kind, context, status, tenant,
object, provider-model, user, duration and token usage; a recent errors table isolates failures.
The underlying events use the sourcetype pattern trackme:ai_assistant:*.
Note
Cancelled chats are deliberately not indexed (the user walked away — counting them would
distort the error rate). Non-entity chat contexts (e.g. Virtual Tenants) are bucketed under a
synthetic system tenant so the by-tenant breakdown stays stable.
The audit envelope¶
All AI audit events share a common envelope so the dashboards can aggregate without parsing JSON. Useful fields include:
Field |
Meaning |
|---|---|
|
The tenant the event belongs to ( |
|
The advisor name (Advisor events), |
|
|
|
|
|
|
|
|
|
The resolved provider account and model. |
|
Wall-clock time and token usage. |
|
Correlates a bridge-launched advisor run with the chat conversation that triggered it. |
Tip
To find which conversation triggered a given advisor run:
index=trackme_summary sourcetype="trackme:ai_agent:*advisor*"
launched_by="ai_assistant" chat_session_id="chat-XXX-yyy"