AI Routines

Overview

AI Routines let you describe a monitoring task in plain English — and TrackMe watches for it on a schedule, then notifies you or acts, autonomously and under your explicit consent.

Where the AI Assistant answers when you ask, and the AI Advisors investigate when you (or the automation) dispatch them, an AI Routine is your own standing instruction: “Tell me if 4 or more high priority feeds turn red within 15 minutes, correlate what they have in common, and add a note to each entity documenting the incident.” You write that sentence once; TrackMe compiles it into a structured instruction, runs it on a cron schedule of your choice, and delivers the outcome — an email or a Splunk event, and (if you granted it) real, audited actions.

Start from a template or from a blank intent — either way, TrackMe shows you exactly how your routine will run before anything is created:

Important

AI Routines share the AI Advisors’ platform requirements: Python 3.13.x (Splunk 10.2.x and later) and at least one configured AI provider. The feature is opt-in by construction — it is completely inert until the AI Assistant is enabled and a provider is configured — and ships with its own kill switch (enable_ai_routines) and a creator-roles policy (administrators only, or administrators and power users) in the system settings.

The three evaluation strategies

Every routine picks how its condition gets evaluated — this is the main cost dial:

• Agentic — the LLM runs on every scheduled cycle and does everything: it reads the tenant, decides whether the condition is met, and acts. The most capable and flexible option (it reasons over live context each run), and the most token-intensive.

• Hybrid — a cheap, compiled SPL trigger runs first on each cycle; the LLM is only invoked when the trigger matches. When nothing is happening, a cycle costs zero LLM tokens. This is the recommended shape for tight schedules at scale.

• Deterministic — the trigger fully computes the condition; the LLM formats and delivers the outcome. The cheapest option for conditions that are precisely expressible in SPL.

The trigger is a cost gate in front of the agent, never a replacement for it — the LLM remains central to understanding, correlating, and explaining.

The strategy is chosen at creation time, and the form adapts to it — the inline explainer states the behaviour and the cost consequence, and the cadence hint under the cron field translates your schedule into runs per day so the cost implication is explicit before you save.

Agentic — the LLM evaluates every cycle

The LLM is the evaluation: on every cron tick it reads the live tenant context, reasons over it, and decides itself whether the condition is met. There is no trigger to author — the plain-English intent is the whole configuration. This is the right choice when the condition is fuzzy or judgement-based (“anything unusual compared to the last runs”), when it needs correlation that is hard to express in SPL, or while you are still shaping what “abnormal” means.

The cost consequence is structural: every scheduled cycle is a full LLM run, whether or not anything is happening. At */30 * * * * that is 48 LLM runs a day — the form says so under the cron field. Tighten the cadence and the cost scales linearly with it.

An Evaluate dry-run makes the mode’s behaviour concrete before you save: the How this routine runs pipeline shows the LLM in the evaluation seat, and the token callout extrapolates the measured per-run cost to your cadence (here ~4K tokens per run ≈ 196K tokens/day at every 30 minutes) — the honest verdict below (Would not fire now, with the reasoning) is the same judgement the model will make on schedule:

Hybrid — a trigger gates the agent

A fast, cheap Splunk search — the compiled trigger SPL — runs first on each cycle; the LLM is called only when that trigger returns rows. You keep the full agentic reasoning (once invoked, the agent investigates, correlates and narrates exactly as an agentic routine would), but you pay for it only when the condition is actually present. A quiet cycle costs zero LLM tokens — the run is recorded as a no-fire and the routine goes back to sleep.

Selecting hybrid reveals two extra fields:

• Compiled trigger SPL — authored by the Evaluate step from your intent (you can refine it); it fires when it returns rows, so the canonical shape ends in something like | stats count | where count>=N.

• Trigger time window — the default -5m / now is right for | trackmegetcoll (it reads current state, unaffected by time); widen earliest if your trigger is an index search on a longer cadence (e.g. -24h, or @d for since-midnight).

This is the cost-efficient choice at scale: a tight cadence (every 5 minutes, say) stays cheap because the vast majority of cycles are trigger-only Splunk searches.

On Evaluate, the pipeline gains the Trigger search stage in front of the LLM (“runs first — skips when it finds nothing (no LLM cost)”), and the token callout switches to the hybrid cost model: you pay the per-run cost per matching cycle, not every tick. In this dry-run the trigger matched, so the evaluation simulated the fire end-to-end (verdict Would fire now, with the would-be notification and act-mode writes described — nothing is actually sent or changed):

Deterministic — the trigger decides

Your trigger fully computes the condition; when it matches, the LLM does the minimum — it turns the trigger’s result into the human-readable notification (and, in act mode, the consented follow-up actions) rather than re-deciding whether to fire. The same trigger SPL and time-window fields apply as in hybrid. The cheapest and most predictable strategy: firing behaviour is exactly your SPL, reviewable and testable like any Splunk search, with the LLM spend limited to the cycles that fire — best for a clear-cut, well-defined condition (“any critical entity red for more than an hour”).

The Evaluate dry-run shows the same gated pipeline — but here the trigger owns the decision outright, and the verdict follows your SPL, not the model’s judgement (in this example the condition window had just lapsed, so the honest answer is Would not fire now even though the entities are still red — exactly what the configured 30-minute window dictates):

Choosing, in cost terms

At the same */30 * * * * cadence (48 cycles/day), the three strategies diverge only in how many of those cycles reach the LLM: all 48 for agentic; only the matching ones for hybrid and deterministic (a quiet day costs zero tokens, a noisy day costs one LLM run per matching cycle — with deterministic runs staying individually smaller, since the agent formats an already-decided outcome instead of investigating). A practical path: start agentic while you shape the intent, then let Evaluate compile the trigger and switch to hybrid when the routine earns a tight schedule — and use deterministic when the condition is precise enough that you want SPL, not the model, to own the firing decision. See Cost transparency for how consumption is surfaced per routine and per run.

Notify or Act

A routine runs in one of two action modes:

• Notify (read-only) — the routine observes and delivers: an email, a structured Splunk event indexed to your summary index, or both at once. Recipients and targets are configured on the routine and delivered deterministically — they never depend on the LLM.

• Act — the routine may also change things: tune thresholds, update priorities, attach notes and labels, manage maintenance windows, retrain ML models, trigger Splunk alert actions (open incidents in your ITSM, notify Slack or PagerDuty), and more.

Act mode is governed by explicit, granular consent:

• You grant permission families — capability-oriented checkboxes such as entities thresholds, entities priority, labels/notes/tags, ML model changes, trigger Splunk alert actions, or execute external MCP tools. Each family exposes exactly its tools to the routine at fire time — nothing more.

• Consent is pinned to what you approved: the exact intent text, the exact set of granted families, and (where applicable) the exact alert-action and external-MCP contracts. Editing any of them invalidates the consent and the routine cannot act again until you re-affirm.

• Every write lands in the entity audit trail with an [AI Agent] stamp, exactly like the AI Advisors.

Switching the mode to Act reveals the consent checkbox and the write-permission families — sixteen capability-oriented grants, each exposing exactly its own tools. Tick only what the routine genuinely needs (least privilege), then affirm the consent:

Note

Two read capabilities are independent of the action mode and enabled by default: read-only Splunk searches (a bounded correlation search the routine can use to investigate) and AI Advisor dispatch (the routine can commission up to two inspect-only AI Advisor investigations per run when a deeper expert diagnosis is warranted). Both are reads — they never require consent, and both can be switched off per routine.

Where to find AI Routines

AI Routines live in two places, matching their scope:

• Tenant Home — routines scoped to that tenant (they read and act on this tenant only).

• Virtual Tenants — global routines that correlate across tenants of your choice.

Global configuration

The feature’s deployment-wide switches live in Configuration → General → Artificial Intelligence:

• Enable AI features — the master switch for everything AI in TrackMe (AI Assistant, every AI Advisor, and AI Routines). AI Routines require it to be on.

• Enable AI Routines — switches the AI Routines feature specifically, independently of the other AI features. When disabled, the AI Routines menu entries are hidden from every TrackMe page, the create / evaluate / run-now REST endpoints reject requests, and any scheduled routine execution is refused. Default: Yes.

• Who can create AI Routines — controls the privilege level required to create routines. The default (Admins and power users) lets holders of the trackmeadminoperations or trackmepoweroperations capability create routines; Admins only restricts creation to trackmeadminoperations. Global (cross-tenant) routines always require an admin regardless of this setting. This applies to creation only — running, editing and deleting an existing routine follow the standard tenant RBAC.

• AI Routines run history retention (days) and AI Routines memory depth (runs) — the two memory levers, covered in Memory between runs below. Defaults: 7 days and 8 runs.

Creating a routine

Creation is a single form — with an important twist: you cannot create a routine blind. The Evaluate step is a mandatory dry-run gate for the LLM-driven strategies.

1. Describe your intent in plain English. Be as specific as you naturally would with a colleague: thresholds, time windows, priorities, what to do when it happens.

2. Pick the evaluation strategy, the cron schedule, the AI provider, and the notification channel (email or Splunk event).

3. For act routines: tick the permission families the routine may use, and affirm the consent.

4. Evaluate — TrackMe runs the routine against your live environment as a read-only dry-run: nothing is created, sent, or changed.

The Evaluate result is where the feature earns your trust before it earns its schedule:

• A compiled instruction — the structured spec the routine will run against (objective, watch condition, signals, notify policy, act policy, assumptions). This is what gets persisted and replayed on every run, so what you approve is exactly what executes.

• A verdict — would the routine fire right now, against your real data, and why.

• A permission-coverage check — if your intent implies an action the granted permissions cannot perform (say, the intent wants notes added but the notes family is not ticked), Evaluate tells you before you create it.

• A cost preview — the tokens this evaluation consumed, extrapolated to your chosen cadence, plus an expandable Raw LLM input panel showing byte-for-byte what is sent to the model.

• A refine loop — reply to the evaluation in plain English (“use a threshold of 20 red entities; ignore orange”) and re-evaluate until the compiled instruction says exactly what you mean.

Tip

Routines understand movement, not just current state. The tenant snapshot the agent reads carries each degraded entity’s time since last state change, and a dedicated tool lets the routine list the exact state transitions (flips) since its previous run — including entities that turned red and recovered in between. Conditions like “N entities turn red within M minutes” are evaluated from the real flip history of your tenant, not guessed.

Email notifications

When a routine’s notification channel is Email (or Email & Splunk event), every fire delivers a formatted email to the recipients configured on the routine. Two principles govern the delivery:

• Recipients are configuration, not AI output. The recipient list and the delivery account are set on the routine and used deterministically — the AI authors the content, never the audience.

• Your email delivery accounts are honoured end to end — the same accounts used by TrackMe’s stateful alerting (transport and security settings, allowed recipient domains with out-of-domain recipients skipped, HTML vs text format preference, custom footer). The form pre-selects your first configured delivery account; the local MTA remains available as the default fallback.

The email arrives as multipart: a branded HTML rendering for modern clients and a plain-text fallback carrying the same content. The HTML shell shows the routine name with the tenant, scope and the AI’s confidence as header rows, followed by the AI-authored summary and evidence rendered from Markdown — headings, bullet and numbered lists, tables, inline code, and section rules all render properly, and the LLM output is HTML-escaped before rendering as a security measure. The subject line is stable and filter-friendly: [TrackMe AI Routine] <routine name> — condition met.

In the example above, the mass-degradation routine fired and the email carries the full incident narrative — the affected feeds grouped by root-cause cluster, per-cluster status and likely root cause, and concrete check-lists for the operator — exactly as the agent wrote it.

Splunk event notifications

When a routine’s notification channel is Splunk event (or Email & Splunk event — both channels at once, each delivered independently), every fire indexes one structured JSON event to the index and sourcetype you configure (defaults: trackme_summary / trackme:ai:routines), with the event source set to the routine id so per-routine searches are trivial.

The event always carries a deterministic envelope — tenant_id, routine_id, routine_alias, scope, plus the AI-authored summary, evidence and confidence — and the targets come from the routine’s configuration, never from the AI. Your SPL, alerts and dashboards built on these events never break because a model phrased something differently.

Custom event fields

The optional Customize event fields panel extends that envelope under a strict contract — you own the schema, the AI only supplies values:

• Static fields — fixed values merged into every event (routing and enrichment constants such as team=siem or env=prod). The AI never sees them.

• AI-filled fields — you declare the field name, a one-line instruction, and optionally a list of allowed values; the AI fills the value when the routine fires. Only the declared names ever reach the event: anything else the model produces is dropped, values are length-capped, and a value outside your allowed list is blanked rather than passed through.

The panel shows a live preview of the event shape as you type (AI-filled values as placeholders), and the Evaluate dry-run renders the real thing — an “Event it would index” block built by the exact same code as production delivery, so what you approve is byte-identical to what a fire indexes.

In the example below, the operator declared an anomalies field with the instruction “Add a dictionary of entities and their anomaly_reason value” — the fired event carries a compact entity: reason; entity: reason digest alongside the envelope, ready for downstream parsing:

Tip

Keep AI-filled values short and typed — a severity, a region, a count, a compact top-5 list. For bulk per-entity data (say, every entity’s full status_message), don’t use an event field: join it at search time instead, deterministically and at zero AI cost:

index=trackme_summary sourcetype=trackme:ai:routines
| join type=left tenant_id [ | trackmegetcoll tenant_id=... component=... | fields object, status_message ]

What runs, as who

Each routine is materialised as its own Splunk scheduled search running on the routine’s cron — no shared orchestrator, full failure isolation, and everything is visible with standard Splunk tooling. Splunk RBAC applies to everything the routine reads and does: the execution identity decides which data the routine can search, which entities it can touch in act mode, and whether the routine survives personnel changes.

Ownership is an explicit choice at creation time. When the tenant has a real service-account owner assigned (Virtual Tenants → three-dot menu → RBAC), the wizard presents the identity as a two-option selector:

• Run as the service account (default, recommended for scheduled routines) — the routine is independent of any personal account: it keeps running if you leave, change roles, or your account is disabled.

• Run as me — the routine uses your own permissions; it stops working if your account is disabled.

When the tenant has no service-account owner (owned by nobody / a system user — good Splunk hygiene for knowledge objects, but a Splunk AI agent cannot execute as the system user), there is no choice to offer: the wizard states it plainly and the routine runs as its creator. Assigning a service account to the tenant later and re-saving the routine moves it over:

Global (cross-tenant) routines always run as their creator — there is no single tenant whose service account could span the scope.

The resolved identity is never hidden: the management view carries an Owner column per routine, so an admin can audit at a glance which identity each routine executes with:

Managing routines

The management view lists your routines with their strategy, mode, provider (switchable in place), schedule state, and last-run outcome. From the row menu you can edit, enable/disable, delete, review the run history, or run now — a fire on demand with live progress:

A run ends in one of two honest outcomes — No fire (the condition is not met; the routine stays quiet, no notification noise) or Fired (with the action taken and the notification status). The run history keeps both, along with per-run duration and token counts:

Note

The run history is a lightweight operational view retained for 7 days. For long-range review, use the Review AI Routines activity dashboard below — the underlying events are indexed to your summary index and follow your own retention policy.

Memory between runs

Routines remember. Every LLM-evaluated cycle ends with the agent writing a state fingerprint — a compact, comparable snapshot it authors itself (for example “DSM red=12/112; firewall cluster amer/apac; NOTIFIED”). On the next cycle, the most recent fingerprints are replayed to the agent as the routine’s memory. This is what powers the anti-noise behaviour you see in practice: notify once when the condition is newly met, stay quiet while the same entities remain red, re-notify when it clears and recurs or when the affected set changes materially.

Because the fingerprint is the agent’s own scratchpad, your intent can lean on it: “track the worst-offending entity across runs and tell me when it changes” works — the agent encodes what it needs to remember into the fingerprint. For deeper history than the recent fingerprints (say, “compare this week’s incident count with last week’s”), phrase it in the intent: every fire’s full narrative is indexed, and the agent can search it with its read-only correlation search like any other Splunk data.

Per-entity memory. The fingerprint is a narrative; entity-level intents are set questions — and those get their own dedicated memory. When a fire concerns specific TrackMe entities, the agent also records their exact entity names with the run (affected_objects). On later cycles, the recent fires are folded into a single deduplicated recently fired entities set with per-entity ages (“entity X fired 32m ago”) and handed back to the agent. This makes intents like “do not re-notify for the same entity within 4 hours” or “only alert on entities that were not part of the last fire” deterministic — decided on exact names, never on fuzzy recall from the fingerprint text. It is also token-frugal by design: quiet cycles and use cases that don’t deal with entities carry no per-entity memory at all.

Two storage layers, should you want to investigate manually:

• Run history (short-lived operational index) — the KV Store collection kv_trackme_ai_routine_runs: one row per evaluated run with the outcome, fingerprint, affected entities, tokens, duration and error. Retained for 7 days by default. This also answers “which runs touched entity X?” directly from SPL:

  | inputlookup trackme_ai_routine_runs | search routine_id="<routine_id>" | sort - cycle_at
  

• Indexed run events (long-range, follows your index retention) — every fire’s full summary and evidence in the tenant’s summary index:

  index=trackme_summary sourcetype=trackme:ai_agent:ai_routine:*
  

Both memory levers are tunable under Configuration → General → Artificial Intelligence for deployments whose intents lean intensively on memory: ai_routines_run_retention_days (default 7, 1–90) controls how long the run rows above are kept, and ai_routines_memory_depth (default 8, 0–24) controls how many recent fingerprints are replayed to the AI on each cycle — each remembered run costs roughly 60 extra prompt tokens on every cycle.

Note

The memory has honest bounds: it is per-routine (routines do not read each other’s fingerprints), it replays the most recent runs only, and hybrid quiet cycles leave no memory — when the trigger does not match, no LLM run happens at all (that is the cost model working as intended), so the memory timeline is “every LLM-evaluated cycle”, not “every cron tick”.

Cost transparency

Token consumption is a first-class citizen throughout: the routine list shows max/avg/last tokens per routine, the run history carries per-run counts, Evaluate extrapolates a daily cost for your cadence, and the dedicated cost view breaks consumption down over time:

A few cost habits that pay off:

• Prefer hybrid for tight schedules — quiet cycles cost zero LLM tokens.

• Right-size the cadence: an agentic routine at */30 runs the LLM 48 times a day whether or not anything happened.

• Use the Raw LLM input panel (Evaluate) to see exactly what your routine sends — and keep the intent focused; it is replayed verbatim on every run.

Auditing routine activity

Everything a routine does is observable. The Review AI Routines activity dashboard (Artificial Intelligence menu) charts runs by routine, outcome, and cadence, and lets you drill into each run — alongside the structured events in the summary index and the per-routine execution logs:

See also Auditing AI activity for the AI-wide audit surfaces, and AI Agents automation for the scheduled advisor batches — AI Routines are user-authored and complementary to those product-defined automations.