DSM — Data Source Monitoring

Data Source Monitoring is TrackMe’s original and most fundamental component. Each entity is a data feed — an (index, sourcetype) combination, optionally split further by a break-by field — and DSM answers the first question of any healthy platform: is my data still flowing, on time, in the expected volume, and well-formed?

What DSM tracks

Signal	What it means
Delay	How old the most recent event is (now − last_event_time). Breached when the latest data is older than the delay max threshold (data_max_delay_allowed).
Latency	The lag between an event’s timestamp and when Splunk indexed it (_indextime − _time). Breached above the lag max threshold (data_max_lag_allowed).
Volume outliers	Unusual event volume, detected by ML density-function modelling — a natural fit for DSM (see Machine Learning).
Future events	Events arriving with a timestamp in the future (beyond tolerance) — usually a time-zone or parsing problem.
Data quality	Unknown formats, mixed formats, and format drift, surfaced by data sampling (see below).

In an entity’s overview you can switch the source of these figures between TrackMe metrics (fast, served from the metrics index) and Splunk queries (direct and exact, but slower and requiring index access).

Tip

Delay and latency are the two most-confused KPIs in TrackMe (people call everything “latency”). If the difference is not crystal clear, read Delay vs latency: what they actually mean — it is short, diagram-led, and worth five minutes.

Defining what gets tracked

• Root constraint — the base SPL filter that defines a tenant’s data scope.

• Allowlist / blocklist — regex rules that include or exclude specific index + sourcetype combinations from automatic entity creation.

• Break-by — an optional extra field that splits one feed into finer entities.

Note

Sourcetype-explosion safeguard. If a single index produces more sourcetypes than the configured cap, bulk DSM creation is refused and an alert is raised. The fix is to blocklist the noisy sourcetypes and re-cover the index with a single <index>:@all aggregate entity.

Delay thresholds, in depth

DSM delay handling is rich, because “late” means different things at different times:

• Static or variable delay. A delay threshold can be a single fixed value, or a variable time-slot policy (day-of-week × hour-of-day) — tighter during business hours, relaxed overnight and at weekends.

• Lagging classes — named, reusable delay/lag rule sets by cadence (high-frequency, nightly batch, quarterly…). Assign a feed to a class for consistent thresholds.

• Adaptive delay — TrackMe can compute delay thresholds automatically from an entity’s own history.

• Threshold lock — lock a feed’s delay and lag thresholds so neither adaptive delay nor a lagging class can rewrite them; a red lock icon marks locked entities.

The full thresholds-and-scoring model lives in Entity State & Scoring.

Note

One authority manages delay per tenant. If the AI Feed Lifecycle Advisor is set to manage DSM delay, it takes over and the legacy adaptive-delay / variable-delay auto-review settings are disabled automatically. Turning the advisor back off does not re-enable them — re-toggle them by hand if you want them back. See the Artificial Intelligence section.

Elastic sources

Elastic sources are virtual DSM entities built from arbitrary SPL — tstats, raw, mstats, or from a data model or lookup — rather than from straightforward index/sourcetype discovery. One search defines one entity. They suit syslog collectors, data-model- or lookup-derived sources, and cases that rely on search-time fields. They come in two flavours: shared (one pooled, concurrency-capped tracker) and dedicated (an independent scheduled report). See Elastic sources for the full treatment.

Tip

For high-cardinality cases, Hybrid trackers have largely superseded elastic sources — but elastic sources remain the right tool for from / raw single-entity edge cases.

Lookup monitoring

DSM can track Splunk lookups (CSV and KV Store) as first-class entities — one entity per lookup — to detect whether a lookup is still being refreshed. Only the delay threshold applies (latency is not meaningful for a lookup). This is covered with the other integrations in Integrations & Remote Deployments.

Data quality: sampling

Data sampling periodically samples a feed’s events and matches them against format models to detect unknown formats, multi-format feeds, and format drift. TrackMe ships out-of-the-box regex models, and you can add your own. Data sampling is configured under operations — see Operations & Administration.

Declaring feeds before they exist

You can declare feeds that should exist from a Splunk lookup — expected sources injection. For DSM both index and sourcetype are required. Injected feeds appear red until real data arrives, then their metadata auto-populates. Imports can be one-shot or recurring.

Coverage-gap analysis complements this: a read-only reconciliation of a reference (a lookup, or a live tstats / raw search) against the feeds TrackMe actually tracks, returning three downloadable lists — only in the reference, only in TrackMe, and in both.

Delayed and inactive entities

When a feed falls so far behind that it drops out of the tracker’s search window, TrackMe does not lose track of it — two cooperating processes keep its delay metadata fresh (this behaviour is shared with DHM):

• Health Tracker — the ``untracked_entities`` task. The tenant’s Health Tracker runs every 5 minutes; its untracked_entities task detects entities no longer covered by any tracker and refreshes their calculated delay, stamping tracker_health_runtime on each entity it handled. Troubleshoot it with:

  index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth task="untracked_entities"
  index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth tenant_id="mytenant" component="splk-dsm" task="untracked_entities"
  

• Delayed entities inspector. A scheduled job named trackme_<component>_delayed_entities_inspector_tracker_tenant_<tenant_id> runs the trackmesplkfeedsdelayedinspector command. It picks up the entities the Health Tracker flagged (via tracker_health_runtime) and runs one targeted search per entity to refresh its metadata, on a tiered cadence keyed to current delay — entities delayed < 24h every ~30 min, < 7 days hourly, and beyond that once every 24 hours — until the entity reaches the tenant’s auto-disablement period. Configure it per tenant under Delayed entities inspector in the tenant settings. The last inspection time is stored on the entity as tracker_runtime (epoch), shown human-readably in the entity’s right-click contextual menu.

Each inspected entity logs a pair of lines — Executing entity delayed tracking resulting search= then delayed_entity_search_results=:

index=_internal sourcetype=trackme:custom_commands:trackmesplkfeedsdelayedinspector tenant_id="mytenant" component="splk-dsm" object="myentity"

You can also follow the inspector (and TrackMe’s other backend logics) in the Handler events tab — the trackme:handlers sourcetype in the tenant’s summary indexes.

See also

• Feeds thresholds (delay & latency) — delay & latency thresholds in full: static, lagging classes, adaptive (ML), and variable delay.

• Expected sources & hosts — in depth — alert on sources that stopped sending.

• Entity State & Scoring — delay, latency, and the impact score in detail.

• DHM — Data Host Monitoring — the host-centric counterpart of DSM.

• Machine Learning — native outlier detection.

• Operations & Administration — data sampling configuration.

• Metrics reference — the DSM delay, latency, and volume metrics.