Configuration Guardian¶

The Configuration Guardian is a built-in framework that continuously checks TrackMe’s own setup and surfaces problems before they cost you data or missed alerts. Each finding is severity-rated and comes with concrete, ordered remediation steps — which both an admin and the AI Assistant can act on directly.

In short:

A modular framework of checks runs continuously in the background (through TrackMe’s scheduled health components), each validating one specific condition — service-account permissions, remote-account connectivity and token expiration, backup freshness, AI-provider reachability, degraded tenants, stale maintenance windows, and more.
When a condition is detected, the Guardian raises an alert carrying a human-readable title, message, remediation guidance and a structured list of recommended actions — actionable both for administrators and for the TrackMe AI Assistant.
Alerts are self-healing: as soon as the underlying condition is resolved, the next detection cycle clears the alert automatically. No manual housekeeping required.

Hint

Where to find it

The Configuration Guardian page is available under Configuration → Configuration Guardian.
Active Guardian alerts also surface as toast notifications on the Virtual Tenants page, each with a View in Configuration Guardian link that deep-links to the alert.

About the checks themselves

The catalog of checks grows over release cycles and each check is fully described in the product itself: the Check Catalog tab documents every check (including the ones that are currently quiet), its severity behaviour, its scheduling host and its tunable parameters. For this reason, this page focuses on the framework and its management features rather than maintaining a per-check list.

Severities, scopes and self-healing¶

Every Guardian check declares:

A scope: tenant (one alert per affected Virtual Tenant) or system (deployment-wide conditions, such as a remote account or a backup issue).
A severity:
- info — a sticky informational notice (for example, the record of an automatic self-healing action). Informational notices do not clear automatically — dismiss them once read.
- warning — attention needed, but the platform keeps working (a fallback or degraded state).
- critical — active degradation: data loss, dropped alerts, or a meta-failure such as the health tracker itself not executing.
An escalation ladder (for probe-style checks): transient conditions start at warning on first detection and escalate automatically to critical once the condition persists beyond a configurable window (24 hours by default for most checks, tunable per check in the Check Catalog). The alert’s metadata tracks since when the condition has been failing.

Findings surface in three places: toasts in the Virtual Tenants UI (coloured by the highest severity present), the REST API (a structured list of active alerts with their recommended actions), and the audit index (every raised, updated and cleared alert). And they self-heal — an alert auto-clears when you fix the underlying condition. Dismissing one in the UI is only temporary: if the condition persists, the next cycle raises it again.

Examples of what it checks¶

A few illustrative checks — the full, current list lives in the in-product Check Catalog:

Check	Severity	Catches
Tenant owner missing a capability	warning	A service account that cannot do its job.
Assigned index does not exist	warning	A tenant index slot pointing at a missing index (or a non-metric metric index).
Remote-account token expiring soon	warning → critical	A bearer token about to expire (see Remote Splunk deployments).
Remote-account connectivity degraded	warning → critical	A remote deployment that has stopped responding.
Backup archive too old / run incomplete	warning → critical	Backups that have gone stale or missed tenants (see Backup & restore).
S3 export failing	warning → critical	An off-site S3 export target that is unreachable, or a run that failed to upload (see Backup & restore).
Health tracker not executing	critical	The meta-check — a tenant’s own health tracker has stopped running.

Detection hosts and cadence¶

Checks are executed by TrackMe’s existing scheduled health components (there is no additional scheduled search to manage):

Per-tenant checks run within each tenant’s health tracker cycles.
System-wide checks and per-tenant meta-checks (checks that must not depend on the tenant’s own tracker — such as detecting that a health tracker is not executing, or that a tenant is operationally degraded) run within TrackMe’s general health manager (daily cadence).
Every check can also be executed on demand at any time (see Running checks on demand).

A special case: inline safeguards (such as the sourcetype-explosion safeguard) detect their condition in real time within the processing hot path, rather than through periodic scans. They are surfaced read-only in the Guardian view — marked with an INLINE SAFEGUARD badge — so administrators get a single must-know list; they can be cleared but not snoozed, disabled or tuned.

Permissions¶

Read-only access to the Configuration Guardian page:

TrackMe user (capability trackmeuseroperations).
Tenant-scoped alerts are RBAC-filtered: users only see alerts for the Virtual Tenants they are allowed to access; system-scoped alerts are visible to everyone accessing the page.

Management actions (snooze, clear, enable/disable checks, tuning, on-demand runs, email notifications): TrackMe admin (capability trackmeadminoperations).

The Configuration Guardian page¶

The page is organized in three tabs, with a KPI strip on top summarizing critical / warning / info counts, active alerts, snoozed alerts and the number of monitored checks.

Active Alerts¶

The alert inbox: every active alert, grouped by check type and ordered by severity, with filters on severity, check type, tenant, scope, and a Show snoozed toggle. On a healthy deployment, the tab simply reports that no alerts match:

Configuration Guardian - no active alerts (healthy deployment)

Each alert renders its full context: severity badge, scope and tenant, the detection message, the remediation guidance, the structured recommended actions, and the raw metadata (Show raw metadata). Per-alert actions:

Clear now — immediately deletes the alert. This is a temporary action by design: if the underlying condition still holds, the alert is re-created on the next detection cycle. Use it to force a re-evaluation, or to acknowledge an informational notice.
Snooze… — silences the alert for a chosen duration (1, 7 or 30 days) with an optional reason (recorded in the audit trail). See Snooze semantics below.
Run this check — re-executes the corresponding check on demand.
Ask AI — opens the TrackMe AI Assistant pre-contextualized on the Guardian, so you can ask for explanations and remediation guidance about the alert (requires the AI Assistant to be configured).

Check Catalog¶

The catalog lists every check the Guardian can fire — including the ones that are currently quiet — so administrators can discover what is monitored without waiting for something to go wrong. For each check:

Description, scope, category, severity behaviour (including the warning → critical escalation ladder when applicable) and scheduling host.
Enable / disable toggle — disabling a check stops its scheduled executions and clears its currently active alerts; re-enabling resumes scanning automatically on the next cycle.
Configure — per-check threshold tuning through a generated configuration modal (for example: token-expiry warning/critical windows, backup-freshness ceilings, escalation windows). Leaving a value at its default keeps the built-in behaviour — only explicit differences are persisted, so future TrackMe default adjustments keep applying to untouched parameters.
Run now and the check’s last run status (when it last executed, by which host, and its outcome).
A search bar to quickly locate a check.

Activity Timeline¶

The timeline is driven by the Guardian’s audit trail (see Audit trail and generated events): per-check counts of created / updated / cleared transitions over a selectable time range, plus a feed of the most recent events. Use it to answer questions like “which check is the noisiest?” or “when did this condition first appear?”.

Snooze semantics¶

Snoozing is the Guardian’s way of saying “I know about this condition, stop showing it to me for a while — but don’t stop watching it”:

Detection never stops. A snooze is a read-time suppression: the check keeps running and the alert record keeps being updated, but the alert is hidden from the toasts and from the default Guardian views until the snooze expires. The Show snoozed toggle reveals snoozed alerts, flagged with a SNOOZED badge, and the KPI strip counts them separately.
Critical breaks through. A snooze remembers the severity it was taken at. If a condition snoozed at warning later escalates to critical, the alert surfaces anyway — flagged BROKE THROUGH SNOOZE — because you silenced a warning, not an active degradation.
Expiry is automatic. When the snooze expires, the alert simply surfaces again if the condition still holds; if it self-healed in the meantime, nothing comes back.
Snoozes are keyed per check and per subject (for example, per tenant), so snoozing one tenant’s alert does not silence the same check on other tenants. An optional free-text reason is recorded with the snooze and lands in the audit trail.

Hint

Clear now vs Snooze vs Disable

Action	Detection	When does it come back?
Clear now	keeps running	immediately on the next cycle if the condition persists
Snooze	keeps running	after the snooze expires (or on critical escalation)
Disable check (Catalog)	stops entirely, alerts cleared	only when the check is re-enabled

Running checks on demand¶

Administrators can force a scan at any time:

Run all checks (page header) — executes every enabled check across every enabled Virtual Tenant.
Run this check — from an alert or from the Check Catalog, optionally narrowed to the alert’s tenant.

On-demand runs from the UI are dispatched as a Splunk search job running the trackmeguardianrunchecks command. This matters on larger deployments: a fleet-wide run can legitimately take several minutes (some checks probe remote accounts and AI providers with generous per-subject timeouts), and running it as a search job guarantees the action cannot be interrupted by web-proxy or load-balancer timeouts.

The same command is available directly from the Splunk search bar:

| trackmeguardianrunchecks

| trackmeguardianrunchecks tenant_id="mytenant" check_type="tenant_operations_degraded"

The command returns one summary row: status, created, cleared, unchanged, skipped and run_time (a failed run yields status="failure" plus a message field). Both arguments are optional; with no filters, every registered check runs against every enabled Virtual Tenant. The command requires the trackmeadminoperations capability.

Email notifications¶

The Guardian can notify administrators by email on alert lifecycle transitions. Click Email notifications in the page header to configure:

Email delivery account — any account configured under Configuration → Emails delivery (or the default localhost MTA).
Recipients — one or more email addresses (the delivery account’s allowed email-domains policy applies).
Minimum severity — warning by default, so informational notices never generate emails; set to critical to only be notified about active degradations.
Events — three independent toggles: New alert created, Alert escalates to critical, and Alert resolved (the condition self-healed).

Notifications are rendered as TrackMe-branded HTML emails (with a plain-text fallback) carrying the alert’s message, remediation guidance and recommended actions; the resolved variant is a compact all-clear.

Guardian email notification - alert resolved

Notification semantics — designed to stay trustworthy

Emails fire on genuine transitions only: routine re-detections of an unchanged condition never notify.
Snoozed alerts never notify — you muted them. A critical alert breaking through a lower-severity snooze does notify, consistently with the UI behaviour.
A resolved email is only sent when the condition actually went away (self-healed). Administrative actions — clearing an alert manually, disabling a check, deleting a tenant — stay silent.
Delivery is fail-open: an SMTP problem can never affect detection; failures are logged and the next transition tries again.

Alert actions integration (advanced)¶

For teams routing notifications through their existing alerting pipelines (email templates, Splunk alert actions, third-party integrations), TrackMe additionally ships an opt-in saved search: trackme_guardian_critical_notifier.

Disabled by default; enable it and attach any Splunk alert action(s) of your choice.
It fires when an alert becomes critical — either created as critical, or escalated from warning to critical — with per-alert throttling so a persistent condition notifies once, not every cycle.

Audit trail and generated events¶

Every Guardian state transition is recorded in the TrackMe audit index (trackme_audit by default, honouring your trackme_audit_idx configuration) under the sourcetype trackme:audit:guardian:

Action	Emitted when
`guardian_alert_created`	a new alert is raised for a (check, subject) pair
`guardian_alert_updated`	an existing alert changes severity or meaningful metadata
`guardian_alert_cleared`	an alert is cleared (self-healed, manually, or via tenant deletion — the `reason` field says which)
`guardian_alert_snoozed`	an administrator snoozes an alert
`guardian_alert_unsnoozed`	an administrator removes a snooze before expiry
`guardian_check_enabled`	a check is re-enabled from the Check Catalog
`guardian_check_disabled`	a check is disabled from the Check Catalog
`guardian_check_config_updated`	a check’s tunable parameters are updated
`guardian_notifications_updated`	the email notification configuration is updated

Each event carries the alert key, check type, scope, tenant, subject, severity (and prior severity on transitions) and title. Routine re-detections of an unchanged condition are deliberately not re-emitted, so the audit trail reflects genuine state changes. Example searches:

`trackme_audit_idx` sourcetype="trackme:audit:guardian" action="guardian_alert_created" severity="critical"

`trackme_audit_idx` sourcetype="trackme:audit:guardian" check_type="<check_type>" | sort - _time

State storage¶

The Guardian persists its state in two global KV Store collections, also exposed as lookups:

trackme_configuration_guardian_alerts (collection kv_trackme_configuration_guardian_alerts): one record per active alert.
trackme_configuration_guardian_settings (collection kv_trackme_configuration_guardian_settings): the management state — per-check enable/disable and tuning overrides, active snoozes, per-check last-run bookkeeping, and the email notification configuration.

| inputlookup trackme_configuration_guardian_alerts

On-demand run logs¶

Executions of the trackmeguardianrunchecks command log to the sourcetype trackme:custom_commands:trackmeguardianrunchecks (index _internal).

REST API¶

The Guardian exposes a complete REST surface under /services/trackme/v2/configuration — listing alerts, running checks, clearing, snoozing, enabling/disabling and tuning checks, and managing the email notification configuration. Refer to the built-in REST API Reference page (API & tooling menu) for the full, self-documenting endpoint catalog; the on-demand scan endpoint is also the recommended integration point for programmatic callers:

| trackme mode=post url="/services/trackme/v2/configuration/admin/run_guardian_checks" body="{}"

AI Assistant integration¶

When the TrackMe AI Assistant is configured:

The Ask AI button (page header and per alert) opens a chat pre-loaded with the Guardian context: the assistant sees the active alerts, their recommended actions and the general knowledge of every check, and can guide you through remediation and verification.
The Virtual Tenants and Tenant Home assistant contexts also include the Guardian state, so questions like “why aren’t my entities updating?” can be answered by cross-checking the relevant Guardian alerts.