FLX (Flex Objects) Use Case Template Catalog

What this page is

The full catalog of every pre-built FLX use case template that ships with TrackMe — 65 templates in v2.3.23, grouped by category, with metrics, schedule, source data, dependencies, and ML-outlier flag per entry.

The page ends with a gap-analysis checklist organised by domain (license, indexer cluster, ingestion, SH workload, good practices, data quality, Cribl, SOAR, host OS, …) — work through it one question at a time when auditing an existing TrackMe deployment.

Catalog version:

TrackMe v2.3.23 (65 templates).

Where the source lives:

package/lib/flx_library/<uc_ref>.json inside the TrackMe Splunk app — one JSON file per template.

Introduction

TrackMe’s Flex Objects (FLX) component turns any Splunk SPL search into a tracked entity with custom KPIs, dynamic thresholds and stateful alerting. Out of the box, TrackMe ships 65 pre-built FLX templates that cover the most common monitoring needs across Splunk Enterprise, Splunk Cloud, Splunk SOAR, Cribl Stream / Cribl Edge, and the canonical Splunk technology add-ons (TA-nix, Splunk_TA_windows).

Every template declares:

• uc_ref — template identifier (matches the filename without .json).

• uc_vendor — broad provenance: Splunk or Cribl.

• uc_category — finer-grained grouping used in the wizard.

• uc_description — one-sentence intent shown before creation.

• uc_earliest / uc_latest — relative time bounds of the SPL.

• uc_cron — schedule for the generated saved-search.

• uc_metrics — flat list of KPI names produced by the search.

• uc_implementation_comments — pre-requisites, caveats, documentation links.

• uc_search — the SPL backing the use case (not reproduced here — see source).

Of the 65 templates:

• 2 vendors are represented — Splunk (54 templates) and Cribl (11).

• 11 categories group them — see the per-category tables below.

• 20 templates opt into native ML outlier detection by emitting an outliers_metrics block in their SPL (flagged yes in the Outlier-friendly? column).

• Schedules range from every 5 min (the majority) through half-hourly, hourly and twice-daily, down to a handful of once-a-day batch searches anchored at quiet hours (license rollups around 03:00 – 04:00 UTC, field-quality at 06:00 UTC).

How to read the schedule column

• */5 * * * * → every 5 minutes

• */15 * * * * → every 15 minutes

• */30 * * * * → every 30 minutes

• 0 * * * * → hourly on the hour

• 0 */12 * * * → every 12 hours (00:00 and 12:00 UTC)

• 5 3 * * * → daily at 03:05 UTC

• 10 4 * * * → daily at 04:10 UTC

• 15 4 * * * → daily at 04:15 UTC

• 0 6 * * * → daily at 06:00 UTC

All schedules run against the tenant’s local time-zone interpretation once dispatched by the Splunk scheduler.

How to read the source-data column

• _internal — Splunk’s own indexed logs (splunkd.log, license_usage.log, metrics.log, …).

• _introspection — host resource usage, KV-store stats, scheduler diagnostics.

• _dsphonehome — deployment-server phone-home events.

• cribl_metrics — Cribl-emitted internal metrics indexed on the Splunk side.

• _cmc_summary / summary — Splunk Cloud monitoring summary events (CMC).

• mgmt API (| rest) — Splunk REST endpoint executed in-search.

• Cribl API (| cribl) — out-of-band call via TA-trackme-cribl.

• SOAR API (| trackmesplksoar) — out-of-band call to a SOAR controller.

• Custom index — anything else (phantom_action_run, lastchanceindex, trackme:fields_quality).

• tstats / mstats — tsidx / metric store; the actual index is parameter-driven.

How to read the dependency column

• none — pure Splunk; runs against indexers / SH out of the box.

• TA-trackme-cribl — required for any | cribl … command.

• Splunk_TA_cribl + cribl_metrics index — required for mstats against Cribl-emitted metrics indexed in Splunk.

• TA-nix — required for cpu_metric / vmstat_metric source-types.

• Splunk_TA_windows — required for Perfmon:* source-types.

• Splunk DB Connect — required to monitor JDBC inputs.

• Splunk App for SOAR / SOAR integration — required for | trackmesplksoar.

• sa-littlehelper — Splunk SA app providing | bundlefiles (Splunkbase #6368).

• TrackMe remote account — when the search is meant to run against a remote Splunk via splunkremotesearch.

See TrackMe Ecosystem — Companion Add-ons for the full companion-add-on inventory.

Catalog by category

cribl_edge

Two templates covering Cribl Edge fleet health, one via the Cribl management API and one via Cribl’s internal metrics indexed inside Splunk.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
cribl_edge_fleet_api	Cribl	Polls Cribl Edge worker nodes via the Cribl management API and alerts if a node stops heart-beating.	cribl.edge.delay_since_heartbeat	every 5 min	Cribl API (| cribl)	TA-trackme-cribl	no
cribl_edge_fleet_metrics	Cribl	Uses Cribl Edge internal metrics indexed in Splunk to compute per-host event-delay and surface silent nodes.	cribl.edge.delay_event_sec	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl + cribl_metrics index	no

cribl_logstream

Nine templates that wrap the canonical Cribl Stream health, traffic and resource metrics. Traffic-flow templates (route / pack / pipeline / total in / out) are outlier-friendly so you get drift detection out of the box.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
cribl_logstream_health_inputs	Cribl	Tracks the percentage of healthy Cribl Stream inputs by input type and group.	cribl_logstream.health.health_inputs	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	no
cribl_logstream_health_outputs	Cribl	Tracks the percentage of healthy Cribl Stream outputs (devnull excluded) and surfaces unhealthy destinations.	cribl_logstream.health.health_outputs	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	no
cribl_logstream_hosts_cpu_usage	Cribl	Watches per-host CPU consumption across the Cribl Stream worker fleet (avg / max / p95).	cribl_logstream.{avg,max,perc95}_cpu_perc	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	no
cribl_logstream_output_destination_pressure	Cribl	Detects blocked and back-pressured outputs that are starving the pipeline.	cribl_logstream.output.{blocked,backpressure}_outputs	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	no
cribl_logstream_pack_traffic	Cribl	Tracks per-Pack event volume in / out and applies ML outlier detection to spot abnormal pack-level traffic.	cribl_logstream.pack.pack_{in,out}_events	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	yes
cribl_logstream_pipeline	Cribl	Tracks per-pipeline event in / out / dropped and their pct sent vs pct dropped to flag silently dropping pipelines.	cribl_logstream.pipeline.{in,out,dropped}_events, pct_{sent,dropped}_events	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	yes
cribl_logstream_route_traffic	Cribl	Per-route in / out bytes and events with ML outlier detection — catches abnormal traffic shifts on a specific route.	cribl_logstream.route.route_{in,out}_{bytes,mbytes,events}	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	yes
cribl_logstream_total_traffic_inputs	Cribl	Aggregate input throughput per input source (bytes + events) with outlier detection.	cribl_logstream.total.total_in_{bytes,events,mbytes}	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	yes
cribl_logstream_total_traffic_outputs	Cribl	Aggregate output throughput per destination (bytes + events) with outlier detection.	cribl_logstream.total.total_out_{bytes,events,mbytes}	every 5 min	mstats on cribl_metrics	Splunk_TA_cribl	yes

splunk_infrastructure

The largest category — 17 templates covering indexer / SH / KV-store / cluster health, parsing quality, queue saturation, and forwarder fleet tracking. This is where most “is Splunk healthy?” alerts live.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
splk_bucket_health_per_index	Splunk	Per-index bucket health — surfaces prematurely-rolled (small) buckets and quarantined buckets (a typical symptom of timestamp parsing failures).	splunk.bucket.{healthy_pct,small_pct,quarantined_pct,total_count,oldest_event_days,index_range_days}	every 30 min	| dbinspect	none	no
splk_bundle_size	Splunk	Knowledge-bundle size per app on the SH tier — a bloated bundle slows search dispatch.	splk.bundle_size_mb	every 15 min	| bundlefiles	sa-littlehelper (Splunkbase #6368)	no
splk_deployment_server_clients	Splunk	Identifies deployment-server clients that have stopped phoning home (silent UF dropout).	ds.time_since_phonehome_sec	every 5 min	_dsphonehome	none	no
splk_dma	Splunk	Data Model Acceleration completeness, size, run duration and bucket count — surfaces stuck DM accelerations with ML outliers.	dma.{complete_pct,size_mb,runduration_sec,buckets_count}	every 5 min	mgmt API (/services/admin/summarization)	none	yes
splk_kvstore_size	Splunk	Per-collection KV-store size against thresholds + total KV-store size.	infrastructure.kvstore.alltime.total_size_mb, size_alert.detected_count	every 5 min	_introspection via /services/server/introspection/kvstore/collectionstats	none	no
splk_kvstore_status	Splunk	KV-store service readiness — alerts when the KV-store goes degraded / down.	(status only, no numeric metric)	every 5 min	mgmt API (/services/kvstore/status)	none	no
splk_large_lookup_files	Splunk	Detects oversized lookup files that slow bundle replication, with outlier detection on total lookup size.	lookups.total_size_mb	every 12 hours	mgmt API (/data/transforms/lookups)	none	yes
splk_parsing_issues_per_component	Splunk	Aggregates WARN / ERROR from LineBreakingProcessor / AggregatorMiningProcessor / DateParserVerbose — the canonical bad-props.conf symptoms.	splunk.parsing.errors_count + dcount of sourcetypes / sources / hosts	hourly	_internal (splunkd.log)	none	no
splk_queues_filling	Splunk	Tracks indexing-pipeline queue fill (parsing, aggregator, typing, indexer) — surfaces backpressure indicating scale issues.	queue.avg_pct_fill_queue, queue.max_queue_fill_percent, …	every 5 min	_internal (metrics.log)	none	no
splk_search_head_activity	Splunk	Search-head scheduler activity (eligible / dispatched / completed / lag / runtime) with ML outliers — catches drift in SH workload.	sh.{actions_triggered,active_historical,active_searches,delayed,dispatched,eligible,max_lag,max_runtime,scheduler_cycles,total_lag,total_runtime,…}	every 5 min	_internal	none	yes
splk_search_head_healthmon_artifact_count_remote	Splunk	Remote SH availability + dispatched-artifact count (jobs count) — designed to be run against every member of an SHC via splunkremotesearch.	sh_mon.{response_time_sec,artifacts_count}	every 5 min	splunkremotesearch	TrackMe remote account (SHC)	yes
splk_splunk_enterprise_cluster_peers_status	Splunk	Per-peer indexer cluster view — bucket %, primary % and bucket balancing deviation, joined with introspection CPU / mem.	splunk_enterprise.cluster_peer.{pct_buckets,pct_primary_buckets,bucket_count,primary_count,pct_buckets_unbalanced_deviation}, system.{avg_cpu_pct,mem_used_pct}	every 5 min	mgmt API + _introspection	none	no
splk_splunk_enterprise_cluster_status	Splunk	Global indexer-cluster availability and peer up / down / unknown breakdown.	splunk_enterprise.cluster.{pct_cluster_availability,peers_count,peers_up,peers_down,peers_unknown}	every 5 min	mgmt API (/services/cluster/manager/status)	none	no
splk_splunk_infra_cpu_used	Splunk	Per-instance average / p95 / max CPU usage from Splunk introspection.	splunk_introspection.avg_cpu_pct	every 5 min	_introspection	none	no
splk_splunk_infra_log_level_variations	Splunk	Counts events per log_level (INFO / WARN / ERROR / FATAL) across the Splunk infra and surfaces abnormal error-rate trends via ML outliers.	splunk.log_level.{info,warn,error,fatal}	every 30 min	_internal	none	yes
splk_splunk_infra_mem_used	Splunk	Per-instance average / p95 / max memory usage from Splunk introspection.	splunk_introspection.avg_mem_pct	every 5 min	_introspection	none	no
splk_splunk_shc_global_status	Splunk	SHC captain-level health flags — maintenance mode, min_peers_joined, rolling_restart, service_ready, kvstore_maintenance.	splunk_shc.{maintenance_mode,min_peers_joined_flag,rolling_restart_flag,service_ready_flag,kvstore_maintenance_status}	every 5 min	mgmt API (/services/shcluster/captain/info)	TrackMe remote account (any SHC member)	no
splk_splunk_shc_members_status	Splunk	Per-member SHC status — surfaces lost / down / pending members.	(status fields, no numeric metric)	every 5 min	mgmt API (/services/shcluster/status)	TrackMe remote account (any SHC member)	no
splk_uf_clients_versions_tracking	Splunk	Inventories UF client versions via tcpin_connections TERM tstats, picks the highest version as target, alerts on % non- compliant.	splunk_forwarders.{total_count,target_version.compliant,target_version.count}	every 5 min	_internal (tstats over tcpin_connections)	none	no

splunk_data_collection

Nine templates focused on the data flowing into Splunk — license volume, event / host drop detection, HEC errors, DB Connect inputs, and basic host tracking.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
splk_dbconnect	Splunk	DB Connect modular-input health — alerts if a JDBC input is disabled.	(state only, no numeric metric)	every 5 min	mgmt API (/configs/conf-db_inputs)	Splunk DB Connect	no
splk_detect_daily_variations_volume_global	Splunk	Daily global license-pool volume with ML outliers — catches abnormal day-over-day spikes.	license.volume_mb	daily @ 04:10 UTC	_internal (license_usage.log)	none	yes
splk_detect_daily_variations_volume_index	Splunk	Daily license volume per index with ML outliers — same as global but per-index granularity.	license.pool_volume_mb	daily @ 04:15 UTC	_internal (license_usage.log)	none	yes
splk_detect_drop_events_count_absolute	Splunk	Hourly per-index event count and distinct host count (absolute window) with ML outliers — catches feed drop-outs.	splunk.feeds.idx.{events_count,dcount_host}	hourly	tstats	none	yes
splk_detect_drop_events_count_rolling	Splunk	Same as absolute but with a rolling time window for near-real-time drop detection.	splunk.feeds.idx.{events_count,dcount_host}	every 5 min	tstats	none	yes
splk_hosts_tracking	Splunk	Simple per-host data-arrival tracking by index + sourcetype — surfaces silent hosts.	delay_event_sec, delta_ingest_time	every 5 min	tstats	none	no
splk_lastchanceindex	Splunk	Detects events landing in lastchanceindex (a sign of routing / parsing misconfiguration).	lastchanceindex.{events_count,sourcetypes_count}	every 5 min	Custom index (lastchanceindex)	none	no
splk_splunk_hec_errors	Splunk	HEC parsing errors broken down by parsing_err with source-IP dcount.	splunk_hec.{errors_count,parsers_in_errors,sources_ip_dcount}	every 10 min	_internal (HttpInputDataHandler)	none	no
splk_volume_ingested_per_day	Splunk	Per-index daily ingested volume (license-based) with ML outliers to catch unexpected daily drops.	splunk.license.daily_volume_mb	daily @ 03:05 UTC	_internal (license_usage.log)	none	yes

splunk_enterprise

Splunk-Enterprise-specific licensing templates — license pool, license volume rolling windows. (The two cluster-status templates splk_splunk_enterprise_cluster_peers_status and splk_splunk_enterprise_cluster_status are tagged with this category in source but documented above under splunk_infrastructure because they share the cluster-health theme — readers should consult either table.)

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
splk_license_pool_usage	Splunk	Per-pool % used and absolute GB used against the licensed pool quota — alerts when a pool approaches saturation.	license.{pool_pct_used,pool_used_gb}	every 5 min	mgmt API (/services/licenser/pools)	none	yes
splk_license_usage_global_splunkenterprise	Splunk	Rolling global license usage at 60 min / 4 h / 24 h horizons (volume + % pool) with ML outliers — designed for near-real-time anomaly detection.	splunk.license.last_{60m,4h,24h}_volume_mb, ..pool_pct_used	every 5 min	_internal (license_usage.log)	none	yes
splk_license_usage_per_index_splunkenterprise	Splunk	Same as global but per-index — pinpoints which index is driving a license spike.	splunk.license.last_{60m,4h,24h}_volume_mb, ..pool_pct_used	every 5 min	_internal (license_usage.log)	none	yes

splunk_cloud

Six Splunk-Cloud-only templates — storage, SVC consumption (global, per app, per consumer service) and rolling license volume.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
splk_license_usage_global_splunkcloud	Splunk	Splunk Cloud global rolling license usage (60 min / 4 h / 24 h) with ML outliers.	splunk.license.last_{60m,4h,24h}_volume_mb	every 5 min	_internal (license_usage.log)	none	yes
splk_license_usage_per_index_splunkcloud	Splunk	Splunk Cloud per-index rolling license usage with ML outliers.	splunk.license.last_{60m,4h,24h}_volume_mb	every 5 min	_internal (license_usage.log)	none	yes
splk_splunk_cloud_storage_usage	Splunk	Splunk Cloud stack storage used vs licensed and % consumption against acceptable thresholds.	splunk_cloud.storage.{storage_used_GB,used_pct,license_GB}	every 5 min	_cmc_summary (splunk-storage-summary)	none	no
splk_splunk_cloud_svc_usage	Splunk	Splunk Cloud stack SVC usage % vs licensed SVC for the whole stack, with ML outliers.	splunk_cloud.svc.{pct_svc_used,utilized_svc,license_svc}	every 5 min	_cmc_summary (splunk-svc)	none	yes
splk_splunk_cloud_svc_usage_by_app	Splunk	Per-app SVC consumption attribution.	splunk_cloud.svc.{pct_svc_used,utilized_svc}	every 5 min	_cmc_summary (splunk-svc-search-attribution)	none	yes
splk_splunk_cloud_svc_usage_by_consumer_service	Splunk	Per-consumer-service SVC attribution (ad-hoc, scheduled, …).	splunk_cloud.svc.{pct_svc_used,utilized_svc}	every 5 min	_cmc_summary (splunk-svc-search-attribution)	none	yes

splunk_soar

Nine templates wrapping the TrackMe SOAR integration — adhoc / playbook action failures, playbook concurrency, asset health, automation-broker HA, infra load / memory, service health, and Splunk-forwarding back into Splunk.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
splk_soar_actions_adhocs_failures	Splunk	Adhoc SOAR action failures (no playbook) — counts total / success / failed.	soar.adhoc.actions_count{,_failed,_success}	every 15 min	Custom index (phantom_action_run)	SOAR-to-Splunk forwarding	no
splk_soar_actions_playbooks_failures	Splunk	Playbook-driven SOAR action failures — counts total / success / failed.	soar.playbook.actions_count{,_failed,_success}	every 15 min	Custom index (phantom_action_run)	SOAR-to-Splunk forwarding	no
splk_soar_assets_health	Splunk	Active asset connectivity check against SOAR assets via the SOAR API.	(status only)	every 30 min	SOAR API (trackmesplksoar)	Splunk App for SOAR integration + automation user	no
splk_soar_automation_brokers_manage	Splunk	Automation-broker HA management — monitors broker pool members and updates assets when a broker goes inactive.	(operational state)	every 5 min	SOAR API	Splunk App for SOAR integration	no
splk_soar_concurrent_playbooks	Splunk	SOAR playbook concurrency by status (success / waiting / running / failed / cancelled) with ML outliers.	soar.playbook_status.{success,waiting,running,failed,cancelled}	every 5 min	SOAR API	Splunk App for SOAR integration	yes
splk_soar_forwarding_splunk	Splunk	SOAR → Splunk forwarding health — counts containers indexed and time-since-last-container, surfacing forwarder breakage.	soar.{containers_count,containers_track_delay}	every 5 min	tstats on phantom_*	SOAR-to-Splunk forwarding	no
splk_soar_infra_load	Splunk	SOAR CPU load (1, 5, 15 min) with ML outliers.	(load metrics)	every 5 min	SOAR API	Splunk App for SOAR integration	yes
splk_soar_infra_memory	Splunk	SOAR memory used / cached / free + percentages with ML outliers.	(memory metrics)	every 5 min	SOAR API	Splunk App for SOAR integration	yes
splk_soar_services_health	Splunk	SOAR services health from the API — surfaces stopped or degraded services.	(status only)	every 5 min	SOAR API	Splunk App for SOAR integration	no

splunk_good_practices

Three templates that flag anti-patterns: searches scheduled with all time earliest, searches owned by a real user instead of nobody, and dynamic sourcetypes (numerically-suffixed foo-1234 style).

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
splk_good_practices_alltime_scheduled	Splunk	Enabled scheduled searches running over all time — a notorious cost driver.	good_practices.splunk.alltime_scheduled	every 12 hours	mgmt API (saved/searches)	none	no
splk_good_practices_detect_dynamic_sourcetype	Splunk	Detects dynamic sourcetypes (regex match on -\d+$) for review.	splunk_good_practices.dyn_sourcetype.{detected_count,total_sourcetype_count,regular_sourcetype_count}	every 12 hours	tstats	none	no
splk_good_practices_run_as_user_scheduled	Splunk	Scheduled searches running on behalf of a named user (vs nobody / service account) — a fragility risk on user offboarding.	good_practices.splunk.runasuser_scheduled	every 12 hours	mgmt API (saved/searches)	none	no

splunk_data_quality

A single template that wraps TrackMe’s fields-quality monitoring approach.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
splk_splunk_fields_quality	Splunk	Aggregates per-(source, sourcetype, host) field-quality check outcomes — % passed / failed and totals; ML outliers for drift.	percentage_{passed,failed}, total_fields_{checked,failed,passed}	daily @ 06:00 UTC	Custom (index=summary sourcetype=trackme:fields_quality)	TrackMe fields-quality plumbing	yes

ta_nix

Two templates leveraging the Splunk TA-nix metric input.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
ta_nix_cpu_usage	Splunk	Per-host avg / max / p95 CPU usage from TA-nix cpu_metric, enriched with OS / IP metadata.	ta_nix.{avg,max,p95}_cpu_usage_pct	every 5 min	mstats on cpu_metric	TA-nix	no
ta_nix_mem_usage	Splunk	Per-host avg / max / p95 memory usage from TA-nix vmstat_metric, enriched with total memory.	ta_nix.{avg,max,p95}_mem_used_pct	every 5 min	mstats on vmstat_metric	TA-nix	no

ta_windows

Two templates leveraging the Splunk TA for Windows Perfmon source-types.

uc_ref	Vendor	What it detects / monitors	Metrics produced	Schedule	Source data	Dependencies	Outliers?
ta_windows_cpu_usage	Splunk	Per-host avg / max / p95 CPU usage from Perfmon:CPU / % Processor Time.	ta_windows.{avg,max,p95}_cpu_usage_pct	every 5 min	mstats on Perfmon:CPU	Splunk_TA_windows	no
ta_windows_mem_usage	Splunk	Per-host avg / max / p95 memory usage from Perfmon:Memory / % Committed Bytes In Use.	ta_windows.{avg,max,p95}_mem_used_pct	every 5 min	mstats on Perfmon:Memory	Splunk_TA_windows	no

Cross-cutting observations

ML outlier coverage

Twenty of the 65 templates pre-wire native ML outlier detection (an outliers_metrics block in their SPL). Concentrated in:

• license / volume / SVC consumption — splk_detect_daily_variations_volume_*, splk_license_*, splk_volume_ingested_per_day, splk_splunk_cloud_svc_usage*,

• Cribl Stream traffic-flow — cribl_logstream_{route,pack,pipeline,total_traffic_*},

• Splunk infra activity — splk_search_head_activity, splk_search_head_healthmon_artifact_count_remote, splk_splunk_infra_log_level_variations, splk_dma, splk_large_lookup_files,

• SOAR runtime — splk_soar_concurrent_playbooks, splk_soar_infra_{load,memory},

• Data-quality drift — splk_splunk_fields_quality,

• Event-count drop detection — splk_detect_drop_events_count_{absolute,rolling}.

These are the templates to keep enabled when you turn on native outlier detection in a tenant. See Outlier detection.

Schedule cadence distribution

• 5-minute cadence — 48 templates (the default for near-real-time health).

• 10 – 15-minute cadence — 3 templates (splk_bundle_size, splk_splunk_hec_errors, the two SOAR action-failure templates).

• 30-minute cadence — 3 templates (splk_bucket_health_per_index, splk_splunk_infra_log_level_variations, splk_soar_assets_health).

• Hourly — 2 templates (splk_parsing_issues_per_component, splk_detect_drop_events_count_absolute).

• Every 12 hours — 4 templates (the three splunk_good_practices + splk_large_lookup_files).

• Daily — 4 templates (the three license-roll-up templates at 03:05 / 04:10 / 04:15 UTC, and splk_splunk_fields_quality at 06:00 UTC).

Important

When enabling a large number of templates simultaneously, watch your SH scheduler load — the 48 five-minute templates dispatched at */5 * * * * will all queue at the same minute mark. Consider staggering by editing uc_cron to use offsets (e.g. */5 * * * * for some, 1-59/5 * * * * for others) at enablement time.

Source-data hot spots

• license_usage.log (_internal) is touched by 6 templates — license is the single most-monitored signal in the catalog.

• _internal and _introspection combined back the bulk of Splunk infra templates — no external add-on needed.

• mstats is the workhorse for all Cribl, TA-nix and TA-windows templates — metric-store performance matters for these.

• | rest against the local management API powers SHC, indexer cluster, KV-store, DMA, lookups, DB-Connect and good-practices templates — these are cheap but require the right capabilities on the running role.

Gap-analysis prompts

Use the catalog above as a checklist. The following questions help you spot coverage gaps you may not have realised you had — work through them one by one when reviewing a tenant’s FLX inventory.

License & cost

• Am I monitoring global license volume with ML outlier detection? (splk_license_usage_global_splunkenterprise / ..._splunkcloud)

• Am I monitoring per-index license volume so I can attribute spikes? (splk_license_usage_per_index_*)

• Am I monitoring license pool % used (Splunk Enterprise)? (splk_license_pool_usage)

• On Splunk Cloud, am I tracking storage usage? (splk_splunk_cloud_storage_usage)

• On Splunk Cloud, am I tracking SVC usage globally, by app and by consumer? (splk_splunk_cloud_svc_usage*)

• Do I have a daily licensing roll-up I can correlate with billing? (splk_volume_ingested_per_day, splk_detect_daily_variations_volume_*)

Splunk indexer / cluster health

• Am I catching SHC member dropout? (splk_splunk_shc_global_status, splk_splunk_shc_members_status)

• Am I catching indexer cluster degradation and balancing deviation? (splk_splunk_enterprise_cluster_status, splk_splunk_enterprise_cluster_peers_status)

• Am I tracking the search head remote artifact count, especially across SHC members? (splk_search_head_healthmon_artifact_count_remote)

• Am I monitoring bucket health (small + quarantined buckets — the timestamp-parsing canary)? (splk_bucket_health_per_index)

• Am I watching queue saturation at the indexer tier? (splk_queues_filling)

• Am I watching KV-store status and size? (splk_kvstore_status, splk_kvstore_size)

• Am I tracking the bundle size on the SH tier? (the silent search- slowdown source) (splk_bundle_size)

• Am I watching lookup file sizes before they blow up bundle replication? (splk_large_lookup_files)

Data collection & ingestion

• Am I catching silent feed drops (event count or host count collapsing)? (splk_detect_drop_events_count_absolute / ..._rolling)

• Am I catching silent host dropout? (splk_hosts_tracking)

• Am I monitoring the last-chance-index for routing / parsing misconfiguration? (splk_lastchanceindex)

• Am I monitoring HEC parsing errors and their source IPs? (splk_splunk_hec_errors)

• Am I monitoring DB Connect modular inputs? (splk_dbconnect)

• Am I monitoring deployment-server clients that stop phoning home? (splk_deployment_server_clients)

• Am I monitoring UF version compliance across my forwarder fleet? (splk_uf_clients_versions_tracking)

• Am I catching parsing-time errors (LineBreaking / Aggregator / DateParser warnings)? (splk_parsing_issues_per_component)

Search-head workload & performance

• Am I monitoring SH scheduler activity (lag, dispatched, eligible)? (splk_search_head_activity)

• Am I monitoring DMA acceleration completeness to catch stuck accelerations? (splk_dma)

• Am I monitoring Splunk log-level variations to catch error storms early? (splk_splunk_infra_log_level_variations)

Splunk good practices

• Am I auditing all-time scheduled searches? (splk_good_practices_alltime_scheduled)

• Am I auditing scheduled searches that run as a real user (offboarding risk)? (splk_good_practices_run_as_user_scheduled)

• Am I auditing dynamic sourcetypes that bloat the metadata cache? (splk_good_practices_detect_dynamic_sourcetype)

Data quality

• Am I running fields-quality monitoring with ML outliers on the CIM-mapped data sources? (splk_splunk_fields_quality)

Cribl Stream / Cribl Edge

• Am I monitoring input and output health % across the Cribl Stream fleet? (cribl_logstream_health_inputs, cribl_logstream_health_outputs)

• Am I detecting blocked or back-pressured outputs? (cribl_logstream_output_destination_pressure)

• Am I tracking route / pack / pipeline traffic shifts with ML outliers? (cribl_logstream_{route,pack,pipeline,total_traffic_inputs,total_traffic_outputs})

• Am I tracking Cribl host CPU usage? (cribl_logstream_hosts_cpu_usage)

• Am I tracking Cribl Edge node heartbeats (API or indexed metrics)? (cribl_edge_fleet_api, cribl_edge_fleet_metrics)

SOAR

• Am I tracking SOAR playbook failures and adhoc-action failures? (splk_soar_actions_playbooks_failures, splk_soar_actions_adhocs_failures)

• Am I tracking playbook concurrency by status? (splk_soar_concurrent_playbooks)

• Am I tracking SOAR asset health with an active connectivity check? (splk_soar_assets_health)

• Am I managing automation broker HA? (splk_soar_automation_brokers_manage)

• Am I tracking SOAR services health (which daemons are alive)? (splk_soar_services_health)

• Am I tracking SOAR infrastructure load and memory? (splk_soar_infra_load, splk_soar_infra_memory)

• Am I tracking SOAR-to-Splunk forwarding (so I notice when SOAR stops shipping containers)? (splk_soar_forwarding_splunk)

Host OS metrics (if you collect them in Splunk)

• Am I tracking Linux CPU / memory for hosts running TA-nix? (ta_nix_cpu_usage, ta_nix_mem_usage)

• Am I tracking Windows CPU / memory for hosts running Splunk_TA_windows? (ta_windows_cpu_usage, ta_windows_mem_usage)

• Am I monitoring Splunk-instance CPU / memory from introspection (separate from the OS-level TA above)? (splk_splunk_infra_cpu_used, splk_splunk_infra_mem_used)

Coverage gaps not addressed out of the box

A few common monitoring needs are not covered by the pre-built FLX library and require either a custom FLX use case (the wizard supports it) or a different TrackMe component:

• Authentication failure spikes (login storms, brute-force) — not in FLX library; build a custom FLX, or rely on Splunk ES.

• Scheduler skipped-search ratios per user / app — partially covered by the WLK component (see WLK — Workload Knowledge).

• Field-extraction performance per sourcetype — handled by the FQM component (see Use Case Demo: Fields Quality (CIM and non-CIM)).

• Saved-search execution failures — handled by WLK, not FLX.

• Splunkbase app version drift — no template; would require a custom FLX pulling /services/apps/local.

• Bucket replication factor / search factor mismatch — partially surfaced by splk_splunk_enterprise_cluster_peers_status, but no dedicated alert.

• Cribl Stream worker node version drift — no template; equivalent of splk_uf_clients_versions_tracking for Cribl would need a custom FLX.

• Splunk Cloud index growth forecasting — no template; storage % is monitored but trend-based forecasting is left to custom searches.

If you spot a recurring need not covered above, the FLX wizard is the supported way to extend the catalog — see FLX — Flex Objects. Contributing a JSON file back to package/lib/flx_library/ makes it available to every TrackMe deployment.

Appendix — source files

All 65 templates live under package/lib/flx_library/<uc_ref>.json inside the TrackMe Splunk app. The JSON schema is shared across templates; consult any file for the full set of fields including uc_earliest / uc_latest time bounds, uc_implementation_comments for caveats and prerequisites, and uc_search for the actual SPL.