Topology Alerts

A topology map answers “what depends on what”. Topology Alerts answers the next question: “tell me the moment this picture degrades”. Any topology view can carry one or more alerts — the topology itself becomes an alerting object, correlating the health of everything the map covers.

The Alerts modal on a topology view with the alert list showing selection, state, last and next run

Why alert on a topology?

Classic per-entity alerting tells you an entity broke. A topology alert tells you your service broke — because you already encoded which entities constitute the service when you drew the map. One alert covers the payment flow end to end; the notification embeds the rendered map, so the recipient sees at a glance which part of the flow is red.

The state-aware engine

Topology Alerts are state-aware, not fire-and-forget. Each evaluation compares the current health of the covered nodes with the alert’s stored state and only acts on transitions:

  • Open — the covered scope degrades (red by default; orange optionally counts): a new alert cycle opens and notifies.

  • Update — the situation changes while open (more nodes affected, fewer, different ones): an update is recorded. No change → no re-notification. No spam by design.

  • Resolve — everything covered returns to health: the cycle closes, and the resolve notification reports how long the incident lasted.

Two further semantics keep the engine honest:

  • Missing nodes (the underlying entity was deleted) are reported as such rather than treated as healthy or broken.

  • Unavailable data (a tenant’s collections temporarily unreachable) freezes the alert’s state instead of transitioning — an evaluation that cannot see clearly never fabricates a resolve or an open.

If your map uses impact propagation, the alert engine honors it: a node repainted by an upstream failure counts as degraded — the alert fires on the same picture the canvas shows, and the notification names the upstream cause.

What an alert covers

Per alert, you choose:

  • the nodes — an explicit selection of nodes on the view, or all nodes — which dynamically includes nodes added to the view later;

  • the trigger states — red always alerts; orange optionally;

  • the schedule — each alert runs on its own cron schedule (default: every 15 minutes), visible with next-run and run-now controls.

The create/edit panel gathers everything in one place: identity and schedule, the orange-as-alerting toggle, the event index, the delivery options — and the node selection, where automatically select all covers every alertable node of the view including nodes added later, while unticking it hands you a filterable per-node checklist:

The edit topology alert panel with alias schedule orange toggle event index and email delivery options The edit topology alert panel with the drilldown root URL and the automatic node selection covering all aggregates

Notifications

Notable events (always)

Every transition (open / update / resolve) emits a notable event with sourcetype=trackme:topology_alerts — searchable, dashboard-able, and available to any downstream pipeline you already run on TrackMe events. Each alert chooses its own event index; leaving it on the deployment default targets TrackMe’s summary index.

Finding the events

You never have to guess where the events landed — the UI takes you straight to them:

  • the Alerts modal banner states the exact index and sourcetype search scope for the current view, and the Events (this view) button opens a pre-scoped Splunk search across all of the view’s alerts;

  • per alert, the Actions menu → Alert events opens the same search narrowed to that alert (alert_id="<alert>").

The Alerts modal with the Events this view button and the per-alert Actions menu including Alert events A Splunk search over the topology alert events showing a resolve transition with items in alert counts and status

Each event is a structured JSON record carrying the full context of the transition: transition (open / update / resolve), status, alert_id and alert_alias, view_id, a stable event_id, and the per-node detail — items_in_alert (with component, state and anomaly reason per node), plus the items_ok_count, items_missing and items_unavailable breakdowns — everything a downstream correlation or ticketing pipeline needs without re-querying TrackMe.

Email (opt-in)

On top of the events, an alert can deliver email through any TrackMe email delivery account:

  • branded HTML in dark or light theme, up to 50 recipients;

  • the topology itself, rendered server-side as an image — with the same node icons as the canvas, colored by the state that triggered the notification, and including the auto-populated members of expanded aggregates with their KPI displays: the recipient sees the same map the canvas shows, not a table;

  • a deep link back to the live view. The link root is auto-resolved from your Splunk Web configuration and can be overridden per alert — for reverse proxies and custom domains;

  • resolve emails report the incident duration.

An alert-opened email with the transition banner items in alert and the embedded rendered topology The alert form delivery section with email account recipients and the topology image theme

Managing alerts

The Alerts modal on each view is the management surface:

  • create, edit, enable/disable and delete alerts;

  • run now for immediate evaluation;

  • run history — the recent runs with status, duration and outcome, so you can see the engine working (and prove a schedule is firing);

  • Ask AI per alert — hand the alert and its recent runs to the AI Assistant for interpretation.

Hint

Creating and managing alerts on a view requires write access to that view (see Administration). If a view is deleted, its alerts are deactivated and remain inspectable by administrators and the alert’s creator for cleanup.

The run-history table with the full lifecycle of transitions emails durations and outcomes