.. _trackme_admin_flx: splk-flx - Creating and managing Flex Trackers ############################################## Introduction to Flex Trackers ============================= **Flex Trackers are created and managed through TrackMe, these are scheduled backend jobs orchestrating the entities discovery and management for the TrackMe splk-flx component:** - The **splk-flx** component stands for Splunk Flex Objects tracking - This component allows tracking any kind of results of a Splunk search, to manage the resulting entities into TrackMe's unique workflow - A Flex tracker can for example be monitoring modular inputs status on remote Heavy Forwarder, monitoring Data Model Acceleration, or your IOT devices, this can be anything adapted to your context - The Flex component expects a certain convention allowing to identity entities and their status, you can automatically define which Key Performance Metrics should be parts of it and even define default Machine Learning models for Outliers detection - A single Tracker can discover and manage a few, or many entities according to the needs - When creating a trackers, the related knowledge objects will be owned by the owner defined at the Virtual Tenant level - TrackMe keeps records of the knowledge objects related to the Trackers, therefore you need to manage its life cycle through TrackMe .. hint:: **Flex Objects Library** - The Flex Objects component comes with a use case library of dozens of pre-built searches for Splunk Cloud, Splunk Enterprise, Splunk SOAR and even for third party product such as Cribl Logstream - The use cases in the library can easily be loaded when creating a new Flex tracker - See: :ref:`trackme_user_guide_soar_monitoring` - See: :ref:`trackme_user_guide_cribl_monitoring` .. image:: img_v2/user_guide_cribl_monitoring/screen13.png :alt: screen13.png :align: center :width: 1200px :class: with-border .. image:: img_v2/flex_trackers/overview.png :alt: overview.png :align: center :width: 1200px :class: with-border Flex Object use cases Library & Tracker creation ================================================ **You can access the use cases Library through the Hybrid tracker creation wizard:** .. image:: img_v2/flex_trackers/lib/screen01.png :alt: screen01.png :align: center :width: 1200px :class: with-border **The user interface also provides a direct link to the TrackMe use cases library provider custom command:** :: | trackmesplkflxgetuc .. image:: img_v2/flex_trackers/lib/screen02.png :alt: screen02.png :align: center :width: 1200px :class: with-border **The Flex wizard opens, start by defining the tracker identifier:** *Note: this step is important as this identifier is used to categorise and group the entities in the user interface, make sure any form of convention that makes sense for you:* .. image:: img_v2/flex_trackers/create/screen2.png :alt: screen2.png :align: center :width: 1200px :class: with-border **Choose the environment target, if the target is remote, a connectivity check is immediately performed:** .. image:: img_v2/flex_trackers/create/screen3.png :alt: screen3.png :align: center :width: 1200px :class: with-border **Define the search logic, the wizard shows the fields convention as well as their detailed usage, and several examples:** .. image:: img_v2/flex_trackers/create/screen4.png :alt: screen4.png :align: center :width: 1200px :class: with-border .. image:: img_v2/flex_trackers/create/screen5.png :alt: screen5.png :align: center :width: 1200px :class: with-border .. image:: img_v2/flex_trackers/create/screen6.png :alt: screen6.png :align: center :width: 1200px :class: with-border **The following example tracks modular input statuses for the Splunk Okta Add-on on a remote Heavy Forwarder:** .. image:: img_v2/flex_trackers/create/screen7.png :alt: screen7.png :align: center :width: 1200px :class: with-border **Another more advanced example which tracks Data Model Acceleration status on a remote Search Head:** .. image:: img_v2/flex_trackers/create/screen8.png :alt: screen8.png :align: center :width: 1200px :class: with-border **Define the cron schedule, if the use case deals with REST related searches, the time quantifiers generally do not matter:** .. image:: img_v2/flex_trackers/create/screen9.png :alt: screen9.png :align: center :width: 1200px :class: with-border **Test and review, for instance with the Okta modular input tracking:** .. image:: img_v2/flex_trackers/create/screen10.png :alt: screen10.png :align: center :width: 1200px :class: with-border **Test and review with the Data Model example, as we specify metrics and outliers metrics we can observe additional information:** .. image:: img_v2/flex_trackers/create/screen11.png :alt: screen11.png :align: center :width: 1200px :class: with-border **Open in search if you wish to review manually the results in the Splunk Search UI:** .. image:: img_v2/flex_trackers/create/screen12.png :alt: screen12.png :align: center :width: 1200px :class: with-border **Once you are happy with the results, you can proceed to the tracker creation:** .. image:: img_v2/flex_trackers/create/screen13.png :alt: screen13.png :align: center :width: 1200px :class: with-border **After the tracker creation, you can execute it now:** .. image:: img_v2/flex_trackers/create/screen14.png :alt: screen14.png :align: center :width: 1200px :class: with-border Managing Flex Trackers ====================== Deleting a Flex Tracker through the UI -------------------------------------- If you want to delete an existing Flex Tracker, this operation must be done via TrackMe. The reason is that the application keeps track of all knowledge objects that were created for a given tenant, to honour various features such as managing the life cycle of the tenant (enabling / disabling, etc) or the life cycle of the tracker itself. **To manage Flex Trackers, click on:** .. image:: img_v2/flex_trackers/manage/screen1.png :alt: screen1.png :align: center :width: 1200px :class: with-border **The user interface shows available trackers and their related objects:** .. image:: img_v2/flex_trackers/manage/screen2.png :alt: screen2.png :align: center :width: 1200px :class: with-border Select one or more trackers to be deleted and proceed, TrackMe will call the related REST endpoint, knowledge objects will be purged and TrackMe will as well clean up the Virtual Tenant records. TrackMe will **not** purge automatically the entities that were discovered and maintained when the originating tracker is deleting, however these won't be maintained anymore. Deleting a Flex Tracker through REST ------------------------------------ **You can delete a Tracker through the following REST endpoint, example in SPL:** :: | trackme mode=post url="/services/trackme/v2/splk_flx/admin/flx_tracker_delete" body="{'tenant_id': 'mytenant', 'hybrid_trackers_list': 'Okta:prod'}" Key Performance Indicators in Flex Trackers =========================================== When creating a Flex tracker, you can leverage any Key Performance Indicator resulting from the Flex search to generate metrics automatically through TrackMe. The following example tracks Data Model Acceleration (DMA) completeness and metrics, in short the logic is the following: - TrackMe orchestrates and executes a Splunk rest search which returns various information per entity (in that case a given Common Information Model) - The information are interpreted by TrackMe as Key Performance Indicators, leading to the generation and ingestion of metrics in the Splunk metric store - Optionally, these Key Performance Indicators are automatically handled via the Machine Learning Outliers detection engine, ML models will be generated and maintained automatically for these KPIs - Generating Key Performance Indicators, and therefore metrics, is optional, in some cases this may not be relevant and totally expected (such as monitoring statuses of modular inputs) *Example, when defining the DMA tracker, we enable various KPIs resulting from the Flex search:* .. image:: img_v2/flex_trackers/kpis/screen1.png :alt: screen1.png :align: center :width: 1200px :class: with-border *To achieve this, all we need is to generate metrics from our resulting SPL query as a JSON object:* :: | eval metrics = "{'dma.complete_pct': " . complete_pct . ", 'dma.size_mb': " . size_mb . ", 'dma.runduration_sec': " . round(runDuration, 2) . ", 'dma.buckets_count': " . buckets . "}" *Optionally, you can choose which of these KPIs will be candidates for ML Outliers detection, and the basic parameters for the lower / upper threshold breached behaviour:* .. image:: img_v2/flex_trackers/kpis/screen2.png :alt: screen2.png :align: center :width: 1200px :class: with-border *This is as well configured via the SPL query, in a resulting JSON formatted object:* :: | eval outliers_metrics = "{'dma.complete_pct': {'alert_lower_breached': 1, 'alert_upper_breached': 0}, 'dma.runduration_sec': {'alert_lower_breached': 0, 'alert_upper_breached': 1}}" *Once created, the Flex Tracker automatically generates and ingests these metrics in the metric store, and start to generate and maintain ML models for the purpose of Machine Learning Outliers detection:* .. image:: img_v2/flex_trackers/kpis/screen3.png :alt: screen3.png :align: center :width: 1200px :class: with-border *Metrics are generated and indexed in the metric index of the Virtual Tenants:* .. image:: img_v2/flex_trackers/kpis/screen4.png :alt: screen4.png :align: center :width: 1200px :class: with-border *Machine Learning models for Outliers detection will be created and maintained:* .. image:: img_v2/flex_trackers/kpis/screen5.png :alt: screen5.png :align: center :width: 1200px :class: with-border