Data sampling¶

Data sampling is TrackMe’s event-format recognition engine. It periodically samples a DSM feed’s events and matches them against format models, so you catch unknown formats, mixed formats, and format drift — a data-quality signal on top of the data-flow signals delay and latency give you. It is DSM-only.

How it works¶

A per-tenant tracker samples selected feeds every ~20 minutes, takes a window of events, and matches each against the configured models. It records the match percentage per model and identifies the major model. If the major model’s match percentage falls below the acceptable threshold, the feed is flagged with a data-sampling anomaly. It detects:

unknown formats — events that match nothing;
multi-format feeds — events matching more than one model;
format drift — the dominant format changed from the previous run.

Models¶

Out-of-the-box models ship for common formats — syslog (BSD and RFC 5424), JSON, Splunk HEC JSON, XML, CSV, key-value, and more.
Custom models are per-tenant regular expressions. An inclusive model says “a match means this is format X”; an exclusive model says “a match means this is explicitly not X”, which is the tool for suppressing false positives.

Acceptance thresholds are set system-wide in the Configuration page.

Note

Small samples are noisy — low-volume feeds may return too few events; widen the window.
Greedy out-of-the-box regex can match events you wanted flagged — add an exclusive model to suppress, and keep custom patterns from overlapping or you get a “multi-format storm”.
Each sample runs a search — stagger cadences across many feeds.