TrackMe Capability × Component Matrix

What this page is

A flat, exhaustive map of every cross-cutting TrackMe capability against the six monitoring components (DSM, DHM, MHM, FLX, FQM, WLK). For each (capability × component) intersection it records:

  • whether the capability applies to that component,

  • the use case unlocked by turning it on (one line each),

  • the configuration touchpoint (UI page, UCC tab, KV collection, vtenant_account field).

Use this page when you know which components a tenant has enabled and want to audit which cross-cutting features are wired in for those components. It pairs naturally with TrackMe Use Cases — Organised by User Intent (which asks the inverse question: given a goal, which component delivers?).

Legend

In every applicability table:

  • Y — fully supported on that component.

  • P — partial support; the cell footnote explains the caveat.

  • — — not applicable on that component (see What does not apply, and why at the bottom of this page).

The six components — recap

For the full description see TrackMe Use Cases Referential — Overview. Short form:

Code

Full name

Tracks

DSM

Data Source Monitoring

(index, sourcetype) feeds. Includes the lookups search mode.

DHM

Data Host Monitoring

Host × sourcetype activity. 2.3.23+ supports breakby_extra_fields.

MHM

Metric Host Monitoring

Hosts × metric category, mstats-based.

FLX

Flex Objects

Custom SPL turned into tracked entities. 65 pre-built templates.

FQM

Field Quality Monitoring

Field parsing quality, CIM compliance.

WLK

WorkLoad Knowledge

Scheduled-search execution health, versioning, orphan, app enablement.

State, scoring, alerting

These capabilities are largely uniform across components — TrackMe was designed so the alerting and scoring experience is the same regardless of what is being monitored.

Capability

DSM

DHM

MHM

FLX

FQM

WLK

Notes

Hybrid impact scoring (total_score)

Y

Y

Y

Y

Y

Y

Same additive formula across all six; per-component weight names.

4-state model (green / orange / red / blue)

Y

Y

Y

Y

Y

Y

Same machine across all six components.

Stateful alerting (open / update / close)

Y

Y

Y

Y

Y

Y

Common trackme_stateful_alert action.

Acknowledgments (sticky / unsticky)

Y

Y

Y

Y

Y

Y

Common ACK collection.

Auto-ACK alert action

Y

Y

Y

Y

Y

Y

Triggered on any saved-search firing.

Notable events alert action

Y

Y

Y

Y

Y

Y

One event per fire; not de-duplicated by TrackMe.

Smart Status investigations

Y

Y

P

Y

P

Y

DSM / DHM / FLX / WLK have explicit per-component investigation templates; MHM and FQM inherit the generic outliers template.

Disruption queue (grace period)

Y

Y

Y

Y

Y

Y

Identical code path across components.

Email delivery (SMTP)

Y

Y

Y

Y

Y

Y

Same SMTP account; same stateful helper.

Embedded 24h charts in alert emails

Y

Y

Y

Y

Y

Y

generate_charts=1 on the alert action.

AI status reports in alert emails

Y

Y

Y

Y

Y

Y

ai_status_report=1 on the alert action.

Free-style REST call alert action

Y

Y

Y

Y

Y

Y

Targets any /trackme/v2/* endpoint.

Use case unlocked (one line per capability):

  • Hybrid impact scoring — Each anomaly type contributes additively, so one critical breach trips red immediately while several minor ones accumulate into orange before tripping critical.

  • 4-state model — One uniform “what colour is my donut” semantic across every component, every UI surface, every downstream consumer.

  • Stateful alerting — One email per real state transition, not one email per evaluation cycle; opens / updates / closes thread together.

  • Acknowledgments — Silence known issues without losing the underlying state signal. Sticky vs unsticky lets you pick auto-expiry.

  • Auto-ACK — Pre-emptively suppress notifications for entities matching a pattern (“ACK for 2 h on any red in this group”).

  • Notable events — Feed TrackMe state into Splunk ES / correlation searches / incident platforms via a notable-index event.

  • Smart Status — When an entity goes red, an investigation search runs once / 24 h and surfaces likely root-cause hypotheses.

  • Disruption queue — Fewer false-positive pages from 30-second blips; alerts only fire after the configured grace duration of continuous red.

  • Email delivery — Notifications land in inboxes correctly threaded with embedded charts, no external image fetches.

  • AI status reports — Recipient sees a 2 – 3 paragraph contextual investigation directly in the email body, not just a link.

  • Free-style REST call alert action — Wire any TrackMe REST mutation (entity CRUD, mode flip, ACK bulk-op) into a saved-search firing without writing a new alert action.

Classification and protection (policies)

This is the largest cluster of cross-cutting features. Several of them are DSM / DHM specific because they relate to delay / latency thresholds.

Capability

DSM

DHM

MHM

FLX

FQM

WLK

Notes

Priority policies (regex / lookup / SPL)

Y

Y

Y

Y

Y

Y

Per-component policy collection.

Tag policies (regex / lookup / SPL)

Y

Y

Y

Y

Y

Y

Multiple matches union.

SLA policies (regex / lookup / SPL)

Y

Y

Y

Y

Y

Y

“Last wins” when multiple policies match.

Labels (catalog-backed coloured chips)

Y

Y

Y

Y

Y

Y

Single catalog per tenant.

Notes (Markdown annotations)

Y

Y

Y

Y

Y

Y

One tenant collection.

Variable delay (time-slot thresholds)

Y

Y

—

—

—

—

DSM / DHM only.

Lagging classes (named delay / lag rule sets)

Y

Y

—

—

—

—

DSM / DHM only.

Adaptive delay (automated threshold recompute)

Y

Y

—

—

—

—

DSM / DHM only. 2.3.23+ also recomputes variable-delay slots.

Variable dynamic thresholds (time-slot KPI thresholds)

—

—

—

Y

—

—

FLX only. Different mechanism from DSM/DHM variable delay.

WLK tenant-default threshold row

—

—

—

—

—

Y

WLK-only fleet-wide fallback (object_id == "default").

Logical groups (blue protection)

Y

Y

Y

Y

Y

Y

Cross-component membership supported.

Blocklist / Allowlist (regex)

Y

Y

Y

Y

Y

Y

Per-component allowlist collection holding both rule types.

Monitoring-time policy

Y

Y

Y

Y

Y

Y

Tenant default + per-entity override.

Use case unlocked:

  • Priority policies — Don’t manually set priority on 5,000 entities; a regex on the index name covers them.

  • Tag policies — Routing labels (team:auth, env:prod) attach automatically and survive entity recreation.

  • SLA policies — Cleanly bucket entities into platinum / gold / silver uptime tiers per regex or lookup.

  • Labels — Visual lifecycle workflow (under-review → in-progress → resolved) with curated colours.

  • Notes — Persist investigation context for the next operator without polluting structured fields.

  • Variable delay — Don’t alert on the expected nightly-batch window; business-hours threshold stays tight.

  • Lagging classes — Manage thresholds once for 40 high-frequency feeds by editing one class instead of 40 entities.

  • Adaptive delay — Persistently-red delay_threshold_breached entities get their thresholds auto-recomputed from history.

  • Variable dynamic thresholds (FLX) — Tighter KPI thresholds during business hours, looser overnight, on the same metric.

  • WLK tenant-default threshold row — Manage skip% / error count / delay thresholds for the whole tenant from one record; per-entity overrides as exceptions.

  • Logical groups — Don’t alert when only 1 of 100 hosts is down; the cluster’s green % protects struggling members.

  • Blocklist / Allowlist — Scope discovery to index=prod_* only; never track sourcetype=.*:debug.

  • Monitoring-time policy — Auto-pause monitoring outside business hours; entities go orange (silent) instead of false-positive red.

Discovery, sampling, and quality

These are mostly component-specific signals — DSM-only sampling features, FQM-only CIM templates, WLK-only signals, and DHM-only extras.

Capability

DSM

DHM

MHM

FLX

FQM

WLK

Notes

Data sampling (regex format models)

Y

—

—

—

—

—

DSM-only — needs raw events.

Elastic sources (external collectors)

Y

—

—

—

—

—

DSM-only.

Lookups monitoring (search_mode=lookups)

Y

—

—

—

—

—

DSM-only; requires TA-trackme-lookupmonitor.

CIM-aware field-quality templates

—

—

—

—

Y

—

FQM-only.

Sourcetype-explosion safeguard

Y

—

—

—

—

—

DSM-only entity-creation hot path.

WLK orphan / versioning / app enablement

—

—

—

—

—

Y

WLK-only signals.

DHM breakby_extra_fields

—

Y

—

—

—

—

DHM hybrid trackers, 2.3.23+, Custom break-by mode only.

ML and advanced trackers

Capability

DSM

DHM

MHM

FLX

FQM

WLK

Notes

Native ML outlier detection

Y

P

P

Y

P

P

Recommended for DSM and FLX (natural fit). Selectively useful on DHM. Not recommended for MHM (sparse / categorical), FQM (slow-moving), WLK (discrete metrics). Defaults: mloutliers_dsm=1 and mloutliers_flx=1; others 0.

Hybrid trackers (custom per-entity SPL check)

Y

Y

Y

Y

Y

Y

One registry per component. Enterprise caps at 8 per tenant; Foundation rejects; Unlimited uncapped.

Replica trackers (one-way entity replication)

Y

Y

Y

Y

Y

Y

Source and target component must match. Local decision maker does not re-evaluate replicated entities.

Use case unlocked:

  • Native ML outlier detection — Get alerted on volume / KPI anomalies without writing thresholds; the model learns the normal pattern with optional weekly seasonality.

  • Hybrid trackers — Add a custom SPL check that contributes to an entity’s score; useful when the default tracker cannot reach the data or you need a domain-specific check on top of the built-in tracker.

  • Replica trackers — Single pane of glass in a central TrackMe across regional TrackMes without running searches on each for every UI refresh.

Integrations

Capability

DSM

DHM

MHM

FLX

FQM

WLK

Notes

Remote Splunk accounts (splunkremotesearch)

Y

Y

Y

Y

Y

Y

Per-tenant tenant_account selects local or a configured remote. Foundation edition does not support remote deployments.

AI Assistant chat (per-entity context)

Y

Y

Y

Y

Y

Y

Entity-level context types exist for all six components.

Splunk SOAR push (containers + enrichment)

Y

Y

Y

Y

Y

Y

Component-agnostic; works on any rows producing entities.

Use case unlocked:

  • Remote Splunk accounts — Monitor any number of remote Splunk deployments from a central TrackMe with HA URLs and automatic token rotation.

  • AI Assistant chat — Ask “why is this entity red?” on the Tenant Home page and get a context-aware LLM answer that knows the entity’s state, policies, recent ACK history, and active Guardian alerts.

  • Splunk SOAR push — Auto-create SOAR containers from TrackMe alerts for runbook-driven remediation; broker-failover safe.

Multi-tenancy and operations

Cross-cutting capabilities that sit above components. They apply uniformly to every component a tenant has enabled.

Capability

DSM

DHM

MHM

FLX

FQM

WLK

Notes

Virtual Tenants (isolation, RBAC, indexes)

Y

Y

Y

Y

Y

Y

Components are enabled per tenant via tenant_components.

Virtual Groups (read-only cross-tenant cards)

Y

Y

Y

Y

Y

Y

tenants_scope carries per-tenant component lists.

RBAC 3-tier (user / power / admin)

Y

Y

Y

Y

Y

Y

Same model regardless of component.

Maintenance mode (runtime alert silencing)

Y

Y

Y

Y

Y

Y

Global or per-tenant; suppresses alerting-path actions.

Maintenance KDB (windows, SLA exclusion)

Y

Y

Y

Y

Y

Y

Scope cascade global / tenant / component / entity.

Bank holidays (SLA exclusion)

Y

Y

Y

Y

Y

Y

Tenant opt-in.

Backup & restore (multi-archive 3.0.0)

Y

Y

Y

Y

Y

Y

Tenant archive carries every component’s tenant collections.

Configuration Guardian

Y

Y

Y

Y

Y

Y

All registered checks are system- or tenant-scoped, not component-specific.

Schema migrations (per-tenant version)

Y

Y

Y

Y

Y

Y

Each migration may touch any component’s collections.

License enforcement (read-only, edition gates)

Y

Y

Y

Y

Y

Y

Foundation gates FLX and WLK creation; Enterprise caps hybrid trackers per tenant.

Configuration touchpoints

Where each capability is configured. UCC refers to the Configuration page tabs driven by globalConfig.json; vtenant_account is the JSON blob inside the tenant record.

Capability

Touchpoint

Hybrid impact scoring weights

Tenant-level: vtenant_account.impact_score_*; entity-level impact_score_weights JSON.

Stateful alerting

Alert-action parameters on the saved search (Alert Editor); state in kv_trackme_common_stateful_alerting_tenant_{tid}.

Acknowledgments

Tenant Home entity → Ack modal; REST /trackme/v2/ack/*.

Auto-ACK / Notable / Free-style REST call / Smart Status

Alert-action parameters on the firing saved search.

Email delivery

UCC emails tab → trackme_emails.conf.

AI status reports in alert emails

Alert-action ai_status_report=1 + ai_provider_name; provider configured under UCC ai_provider.

Disruption queue

System default in trackme_settings.conf [splk_general]; per-tenant vtenant_account.default_disruption_min_time_sec; per-entity disruption_min_time_sec.

Priority / Tag / SLA policies

Tenant Home per-component “Policies” modal; REST /trackme/v2/splk_<priority|tag|sla>_policies/*.

SLA classes

System-level trackme_settings.conf [sla] (UCC sla tab).

Labels (catalog + assignments)

Tenant Home “Manage Labels” modal; REST /trackme/v2/labels/*.

Notes

Entity overview “Notes” panel; REST /trackme/v2/notes/*.

Variable delay (DSM / DHM)

DSM / DHM entity → “Variable delay threshold” modal; REST /trackme/v2/splk_variable_delay/*.

Lagging classes (DSM / DHM)

Tenant Home DSM / DHM “Lagging classes” modal; REST /trackme/v2/splk_lagging_classes/*.

Adaptive delay

Per-entity allow_adaptive_delay; tracker trackme_<dsm|dhm>_adaptive_delay_tracker_tenant_<tid>.

FLX / FQM / WLK dynamic thresholds

Tenant Home per-component “Thresholds” modal; REST /trackme/v2/splk_<flx|fqm|wlk>/*.

Logical groups

Tenant Home per-component “Logical groups” modal; REST /trackme/v2/splk_logical_groups/*.

Blocklist / Allowlist

Per-component wizard / feed configuration; REST /trackme/v2/splk_blocklist/*.

Monitoring-time policy

vtenant_account.monitoring_time_policy; per-entity override.

Data sampling (DSM)

DSM tab → “Data sampling” modal; REST /trackme/v2/splk_data_sampling/*.

Elastic sources (DSM)

DSM “Elastic sources” wizard; REST /trackme/v2/splk_elastic_sources/*.

Lookups monitoring (DSM)

Hybrid Trackers create wizard, Step 1 search_mode=lookups; requires companion TA TA-trackme-lookupmonitor installed.

WLK orphan / versioning / app enablement

WLK wizard + per-app kv_trackme_wlk_apps_enablement_tenant_{tid}.

Sourcetype-explosion safeguard

trackme_settings.conf [splk_general] sourcetype_max_cap_per_index.

Native ML outlier detection

Tenant Home per-component “Outliers rules” modal; tenant-level vtenant_account.mloutliers_<component>; defaults in trackme_settings.conf [splk_outliers_detection].

Hybrid trackers

Tenant Home per-component “Hybrid Trackers” modal / wizard; REST /trackme/v2/splk_hybrid_trackers/*; license-edition gated.

Replica trackers

Virtual Tenants “Replica Wizard”; REST /trackme/v2/splk_replica_trackers/*; executor trackme_replica_executor saved search.

Remote Splunk accounts

UCC account tab → trackme_account.conf; Remote Accounts Overview page.

AI Assistant (provider config)

UCC ai_provider tab → trackme_ai_provider.conf; master switch trackme_settings.conf [trackme_general] enable_ai_assistant.

AI Assistant (per-tenant)

vtenant_account.ai_provider_name, ai_anonymize_entity_names.

Splunk SOAR

vtenant_account.soar_*; REST /trackme/v2/splk_soar/*; commands trackmesplksoar and trackmesplksoarlookup.

Virtual Tenants

UCC vtenants tab + Configure tenant modal (per-tenant 3-dot menu); REST /trackme/v2/vtenants/admin/*.

Virtual Groups

Virtual Tenants page Create Virtual Group wizard; REST /trackme/v2/virtual_groups/*.

RBAC (per-tenant roles)

Splunk roles (trackme_<admin|power|user>_<tid>) + tenant record’s rbac_<admin|power|user>_roles fields.

Maintenance mode

Maintenance Mode page; REST /trackme/v2/maintenance/*; tracker trackme_maintenance_mode_tracker.

Maintenance KDB

Maintenance KDB page; REST /trackme/v2/maintenance_kdb/*.

Bank holidays

Manage Bank Holidays page; REST /trackme/v2/bank_holidays/*; per-tenant opt-in vtenant_account.bank_holidays_exclusion and bank_holidays_countries.

Backup &amp; restore

Backup &amp; Restore page; REST /trackme/v2/backup_and_restore/*; scheduler trackme_backup_scheduler.

Configuration Guardian

Surfaced as toasts on Virtual Tenants page; REST /trackme/v2/configuration/admin/guardian_alerts, dismiss_guardian_alert, run_guardian_checks.

Schema migrations

Per-tenant schema_version field; runs at tenant create / upgrade / load.

License

License page; REST /trackme/v2/licensing/*; watermark in trackme_settings.conf (HMAC-SHA256).

What does not apply, and why

DSM-only capabilities

  • Data sampling — Inspects raw events; needs index + sourcetype + _raw. DHM works off ingest metadata (no raw text), MHM is metric data, FLX / FQM / WLK build their own metric shape.

  • Elastic sources — A DSM entity is the right granularity for “is this external collector producing data?”.

  • Lookups monitoring — Lookup files / KVstore collections are configuration artefacts, not data feeds; DSM is the closest model. Operates with neutralised latency (no _indextime exists for lookups) and a single delay-threshold alerting hook.

  • Sourcetype-explosion safeguard — Bulk DSM creation discovers an (index, sourcetype) matrix that can blow up if a pipeline mis-routes events. Other components do not pivot on this discovery axis.

DSM + DHM only (not MHM / FLX / FQM / WLK)

  • Variable delay — Day-of-week × hour-of-day thresholds make sense for data feeds and per-host streams that have legitimate cadence variation. MHM uses static thresholds.

  • Lagging classes — Same rationale; only meaningful for delay / latency-based components.

  • Adaptive delay — Recomputes data_max_delay_allowed (and variable-delay slot thresholds since 2.3.23) from historical metrics. Only meaningful for components with a single delay axis to recompute.

WLK-only signals

  • Orphan, versioning, app enablement — They describe properties of a Splunk scheduled search and its hosting app. Not applicable to data feeds, hosts, custom SPL trackers, or field-quality entities.

  • Tenant-default threshold row (``object_id == “default”``) — WLK-specific fallback for “manage thresholds once for the whole tenant, override per entity as exceptions”.

FQM-only

  • CIM-aware templates — FQM is purpose-built for “is this CIM data-model field being parsed correctly?”. The other components do not evaluate per-field coverage / success metrics.

DHM-only (2.3.23+)

  • ``breakby_extra_fields`` — Extra metadata dimensions on top of (index, sourcetype) for DHM hybrid trackers in Custom break-by mode. Specific to how DHM groups per-host activity.

FLX-only

  • Variable dynamic thresholds (time-slot KPI thresholds) — FLX supports day-of-week × hour-of-day slots on its dynamic threshold evaluator. This is a different mechanism from DSM / DHM variable delay — variable delay applies to feed delay thresholds, variable dynamic thresholds apply to any FLX KPI.

Outlier-detection recommendation

DSM and FLX are recommended (event volume / numeric KPIs are natural density-function candidates). DHM is selective (“per-host volume can be noisy”). MHM, FQM, WLK are not recommended:

  • MHM — Metric streams are too sparse or too categorical for density modelling.

  • FQM — Field-quality metrics are slow-moving and regime-dependent; thresholds are the right tool.

  • WLK — Scheduler metrics are discrete (skip count, error count); again, thresholds are better.

Smart Status template coverage

Smart Status investigation templates are documented for DSM, DHM, FLX, WLK anomalies (delay, future data, latency, hosts dcount, inactivity, threshold, skip, error, orphan). MHM and FQM fall through to a generic outliers template — marked P in the matrix.

Replica trackers and the decision maker

Replica trackers replicate an entity set but the local decision maker does not re-evaluate them — it trusts the source’s state. So while the matrix shows Y across all six components (replica trackers work on any component), the surrounding capabilities (logical groups, disruption queue, policies) only apply to non-replicated entities in that component; on replicated entities they are inert by design.

License-edition gating

  • Foundation edition supports DSM / DHM / MHM only; FLX and WLK rows in the matrix are still Y (the capability exists), but Foundation tenants cannot create FLX / WLK entities.

  • Enterprise caps hybrid trackers at 8 per tenant; Foundation rejects hybrid-tracker creation; Unlimited is uncapped.

  • Read-only mode (license expired) disables creation / editing across all six components without removing the capabilities themselves.