Large-scale environments

TrackMe is designed to scale to many tenants and large entity counts. This page collects the practices that keep it efficient as it grows.

Design principles

  • Tenants are the scaling unit. Design them around perimeters, teams, and permissions. Document your index naming conventions and top-priority perimeters first, then map tenants onto them.

  • Prefer many narrow trackers over a few broad ones. Several concurrent hybrid trackers with non-overlapping scope are more cost-efficient, and easier to reason about, than one sweeping tracker.

  • Onboard progressively. Start a tenant with DSM only, then add hybrid trackers one at a time, benchmarking each, rather than enabling everything at creation.

  • Lead with DSM. Data Source Monitoring delivers the most value first; add DHM (heavier, needs maturity) and MHM (IT-Ops-specific) as you grow.

  • Use a service account, not admin, so you can apply Workload Management and attribute cost cleanly (see Configuration).

Important

Dedicate a search-head tier to TrackMe at scale. On-premises, give TrackMe its own SH tier; on Splunk Cloud, use the ad-hoc tier. TrackMe must never actively track on the same search-head tier as a Splunk premium app such as Enterprise Security or ITSI — that configuration is not supported. Use remote deployments to reach data on other tiers.

How the heavy backends run: batches under a runtime budget

When you review TrackMe in the Splunk scheduler, the per-entity backends — adaptive delay, ML training, ML detection, per-entity maintenance — all follow the same execution model, and misreading it is the single most common source of confusion in a workload review:

the dispatch window is a trigger, not a bound. A tracker scheduled every 5 minutes with a -5m window is not promising to finish in 5 minutes, and a multi-minute run is not a runaway search. On each occurrence, the saved search selects a batch of entities, does the real work in a per-entity processing loop, stops when it nears its runtime budget, and resumes with the next batch on the following cycle.

Batches under a runtime budget Three scheduled cycles on a timeline. Each run processes a batch of entities, stops at its runtime budget, and the next cycle resumes with the remaining entities. t run 1 entities 1–120 ⏱ stops at budget t + 5m run 2 entities 121–240 ⏱ stops at budget t + 10m run 3 entities 241–300 ✓ batch done early One estate, processed as consecutive batches — each run resumes where the last stopped

Two practical consequences:

  • Run time reflects batch size, not the window. The more entities a backend has to process each cycle, the longer its runs — that is expected behaviour for a large tenant, not a fault.

  • The cost is self-limiting. These backends only pick up entities that need work (adaptive delay only reviews entities actually in a delay breach; ML training only fits models past their retrain interval). The cleaner and more stable the estate, the smaller the batches — the cost scales with how much of the estate is unstable.

See adaptive delay and the ML backend & scheduling for what one run of each actually does.

Reading the Splunk scheduler: continued vs skipped

When reviewing TrackMe’s searches in the scheduler (index=_internal sourcetype=scheduler), two statuses matter and they are not equivalent:

  • continued — the scheduler could not dispatch the search at its slot because the concurrent-search limit was reached, so it deferred and re-attempted it on the next cycle. The run is postponed, not lost. It is a saturation symptom of the stack, not a TrackMe fault — and TrackMe is resilient to it by design, re-attempting its own work rather than failing the run.

  • skipped — the scheduler gave up on that occurrence. This is the status that genuinely drops work and deserves investigation.

Tip

Give TrackMe a bounded slice of concurrency. Run each tenant under a dedicated service account as the tenant owner, and place role-based concurrent-search limits on that account — restrictive enough that TrackMe cannot crowd out business or premium-app searches, but with enough headroom to operate (on the order of 10 concurrent searches). TrackMe then consumes a predictable slice of the stack instead of competing on the shared pool. This is good Splunk hygiene independent of any footprint concern.

To measure completion, skipping, run times, and TrackMe’s share of compute objectively, see Measuring TrackMe’s scheduler footprint — current TrackMe releases also bundle these scheduler reports in the Support - generate diags tooling.

Reducing the footprint: which levers, in which order

When a workload review asks for reductions, apply the levers in value order:

  1. Upgrade to the current release. The structural gains live there (native ML engine, caching and cadence improvements) — most later levers work better after it.

  2. Disable optional convenience backends you don’t rely on. Adaptive delay is the classic example: it is valuable but entirely optional, and disabling it at the tenant level automatically un-schedules its backend search — thresholds simply hold their current or policy-defined values (see Adaptive delay (ML)).

  3. Scope ML outliers to where behavioural change is meaningful — rather than a blanket on/off (see Outlier detection).

  4. Align cadences to the current defaults. Long-lived tenants can carry legacy schedules: TrackMe deliberately never rewrites an existing saved-search schedule on upgrade, so re-aligning custom crons to today’s defaults is a quick manual win.

Note

Do not reduce the ML-monitor frequency to save cost — that opens scoring gaps and delays detection. The correct ML cost lever is model scope (which entities carry models), not monitor cadence.

Note

Per-entity maintenance, data sampling, and Guardian checks all add scheduled load — stagger their cadences, and keep an eye on skipped searches (see Managing tenants).

See also