Flex Objects - Adapt TrackMe to any monitoring use case (splk-flx)

Flex Objects: The Ultimate Tracking Component

• Flex Objects (splk-flx) is one of TrackMe’s most powerful components, designed to transform the results of any Splunk search into fully tracked, monitored, and alerted entities.

• Unlike standard tracking components that follow a predefined schema, Flex Objects lets you define your own search logic, your own entities, your own KPIs, and your own anomaly detection rules.

• Flex Objects comes with a library of 60+ pre-built use cases covering Splunk infrastructure, data quality, license management, Splunk Cloud, SOAR, Cribl, and host monitoring - ready to deploy in minutes.

• Enterprise Edition & Unlimited Edition feature - Flex Objects is available to Enterprise and Unlimited edition customers.

• Flex Objects is particularly efficient for Outliers detection, high-scale scenarios with thousands of entities, and custom grouping strategies across any kind of data in Splunk.

Hint

Key references

• For a deep dive into the administration of Flex trackers, see splk-flx - Creating and managing Flex Trackers.

• For Machine Learning Outlier detection concepts, see Outliers Anomaly Detection.

• For specific use case implementations, see the dedicated white papers referenced in Real-World Use Cases.

How Flex Objects Works

The Concept

Turn any Splunk search into fully tracked, monitored, and alerted entities.

Every Splunk environment is unique - different data sources, different infrastructure, different monitoring requirements. TrackMe’s standard tracking components (splk-dsm for data sources, splk-cim for CIM compliance) cover the most common scenarios, but there will always be use cases that don’t fit a predefined mold.

Flex Objects is the bridge between your Splunk search expertise and TrackMe’s monitoring, alerting, and incident management capabilities. If you can write a Splunk search, you can create a Flex tracker.

✏️ 1. Define

Write any SPL search or pick from 60+ pre-built templates

⏰ 2. Schedule

TrackMe runs the search on a cron schedule

🔍 3. Discover

Each result row becomes a tracked entity

📊 4. Track

KPIs, metrics, and ML Outlier detection

🔔 5. Alert

Stateful alerting, emails, and active commands

Define or Select

Two paths to get started - pick a pre-built use case or write your own search.

Path 1: The Library - Flex Objects ships with 60+ pre-built use case templates organized by vendor and category (Splunk Infrastructure, Data Quality, License, Cloud, SOAR, Cribl, Host Monitoring). Each template includes a complete Splunk search, pre-configured KPIs, and outlier detection rules. Select a template, customize the parameters to your environment, and deploy in minutes.

Path 2: Custom Search - Write any Splunk search that produces results following the Flex search contract. At minimum, each result must have an object (entity name) and a status (1=green, 2=orange, 3=red). Add optional fields for KPIs, descriptions, and ML anomaly detection as needed.

📚 From the Library

60+ templates across 8 categories

Filter by vendor and category, preview the full search logic, customize and deploy.

🔧 Custom Search

Any SPL producing entities

Use tstats, | rest, mstats, custom lookups, or any SPL command - if it returns rows, it can be a Flex tracker.

Tracker Types

Two types of Flex trackers for different monitoring needs.

When creating a Flex tracker, you choose between two fundamentally different approaches depending on your goal:

🎯 Use Case Tracker

Entity discovery from a Splunk search

Your search produces one row per entity. Each entity is tracked individually with its own status, KPIs, metrics, and outlier detection. This is the standard Flex tracker for monitoring infrastructure, services, data quality, and more.

📈 Converging Tracker

Aggregated availability from existing entities

Correlates multiple Flex entities from one or more tenants, aggregating them into a single converging entity represented as a percentage of availability. Ideal for service-level monitoring and executive dashboards.

TrackMe Orchestrates

Once created, TrackMe takes over the heavy lifting.

Your Flex tracker becomes a scheduled report that TrackMe executes automatically. On every run:

• The Splunk search is executed against the target deployment (local or remote)

• Each result row is matched to an existing entity or discovered as a new one

• Entity states are persisted in the KVstore with full history

• KPIs defined in the metrics field are extracted and stored

• Metric events are ingested into the Splunk trackme_metrics index for long-term trending

• Entities that stop appearing are tracked for inactivity based on configurable thresholds

TrackMe automatically randomizes cron schedules to distribute load, handles batch KVstore operations for performance at scale, and manages the full entity lifecycle from discovery to deletion.

Metrics & ML

Track any number of KPIs per entity with built-in Machine Learning anomaly detection.

The metrics field in your Flex search defines Key Performance Indicators for each entity. TrackMe tracks these over time and ingests them as Splunk metric events, enabling trending, dashboards, and alerting on any KPI.

The outliers_metrics field activates Machine Learning Outlier detection on selected metrics:

• TrackMe builds a statistical model learning the normal behavior of each metric

• Dynamic upper and lower thresholds are calculated automatically

• Time factors add seasonality awareness - the model knows that Monday 9 AM looks different from Sunday 3 AM

• Anomalies are detected when current values fall outside the predicted range

📊 KPIs

Track CPU, event counts, delay, completion %, size, or any numeric metric per entity

🔮 Outlier Detection

ML models learn normal behavior and alert on anomalies - no static thresholds needed

📆 Seasonality

Time factors (hour, day+hour, weekday+hour) ensure the model respects natural patterns

Incidents & Workflow

Flex entities are first-class citizens in the TrackMe Stateful Alerting workflow.

When a Flex entity changes status (green to orange/red, or an outlier is detected), TrackMe’s Stateful Alerting engine manages the full incident lifecycle automatically:

• Incident creation - a new incident is opened and persisted in a state-aware manner

• Incident updates - as the entity condition evolves, the incident is updated with new context

• Incident closure - when the entity returns to a healthy state, the incident is automatically closed

• Rich HTML email notifications - with embedded metrics charts for immediate visual context

• AI-generated status reports - optional AI-driven analysis included in email notifications, adapted to the incident lifecycle stage (opened, updated, closed)

• Active commands - execute generating or streaming commands on incident open, update, or close for advanced integrations (ticketing, webhooks, etc.)

• SLA tracking - measure entity availability over time for compliance reporting

Flex entities also benefit from all TrackMe features: priority management, maintenance windows, logical groups, tags, status flipping analysis, audit changes, and the AI Assistant for intelligent investigation. See Alerting Architecture & Third-Party Integration for full details.

Hint

For detailed administration documentation including the search contract reference, creating trackers step-by-step, KPI configuration, ML Outlier detection setup, and managing trackers, see the splk-flx - Creating and managing Flex Trackers guide.

Real-World Use Cases

Flex Objects powers a wide range of monitoring scenarios. The following white papers provide in-depth, step-by-step implementations:

White Paper	Description
Use TrackMe to detect abnormal events count drop in Splunk feeds	Detect abnormal drops in event counts using Flex Objects with ML Outlier detection. Covers both rolling and absolute metric approaches.
Analyse log messages logging level to detect behaviour anomalies using TrackMe’s Flex Object and Machine Learning Anomaly Detection	Monitor Splunk internal log levels (INFO, WARN, ERROR, FATAL) with Flex Objects and detect unusual spikes using ML.
Tracking Splunk Cloud SVC consumption in TrackMe	Track Splunk Cloud SVC consumption at global and per-app levels using Flex Objects templates from the library.
Monitor Splunk Indexer Clusters	Monitor Indexer Cluster health, peer status, and bucket balance using Flex Objects with remote deployment accounts.
Monitor Splunk Search Head Clusters	Monitor Search Head Cluster global status, member health, and search activity using Flex Objects.
Use Case Demo: 360 Services Monitoring with TrackMe	A comprehensive 360-degree monitoring approach combining Flex Objects with other TrackMe components for full-stack visibility.

Summary

Flex Objects is the key to unlocking TrackMe’s full potential for any monitoring use case. Whether you’re starting from the pre-built library or writing custom searches from scratch, the component provides:

• A simple search contract - just object and status to get started, with optional metrics and outliers_metrics for advanced monitoring

• 60+ ready-to-use templates - deploy pre-built use cases for Splunk infrastructure, data quality, license, cloud, SOAR, and Cribl monitoring

• ML-powered anomaly detection - detect anomalies in any KPI with configurable seasonality awareness

• High-scale performance - optimized for environments with thousands of entities

• Converging visibility - aggregate entity health into service-level availability metrics

Combined with TrackMe’s alerting framework, virtual tenants, and SLA tracking, Flex Objects makes it possible to build a comprehensive, data-driven monitoring platform on top of Splunk.

For step-by-step administration instructions, see the splk-flx - Creating and managing Flex Trackers guide.