QUICK START - Starting with TrackMe: (feed tracking quickstart)

Starting with TrackMe!

• This tutorial is a starting point for TrackMe new joiners!

• The objective is to help you getting started with TrackMe, understand the basic concepts and focus on feeds Tracking for Splunk.

• In this tutorial, we will assume that you are starting entirely from scratch, and we will cover the essential steps to set up feed tracking in TrackMe, with main best practices.

• This tutorial works in 3 main steps and can be used by all users, including Free community users:

  • Step 1: Create a Virtual Tenant for data sources tracking

  • Step 2: Create a Virtual Tenant for data host sources

  • Step 3: Alerting and notifications

TrackMe is installed, what now?

Once TrackMe is installed, the following wizard will guide you through the initial setup of a new Virtual Tenant:

About Virtual Tenants:

• Virtual Tenants are an essential core feature in TrackMe, this allows TrackMe to orchestrate and manage TrackMe related knowledge objects and configuration, and address many powerful concepts such as multi-tenancy, data isolation, and more.

• You can also understand this concept as virtual instances of TrackMe within TrackMe, which you can create, destroy, and restart as needed.

• Virtual Tenants enable TrackMe components, which are designed to address use cases, such as splk-dsm which is the essential component for feed tracking.

Step 1: Let’s create a Virtual Tenant for data sources tracking

We will now create a Virtual Tenant to start tracking data sources:

• component: Splunk Feeds Tracking

• tenant name: feeds-tracking

• tenant alias: This is optional and can be updated at any time. If set, the alias replaces the tenant identifier in the Virtual Tenants UI.

• tenant description: This is optional and can be updated at any time. If set, the description is displayed in the Virtual Tenants UI.

Creating tracker during the Virtual Tenant creation, or later on via the Hybrid Tracker management UI

• The next 3 screens of the wizard for feed tracking are about the configuration of the TrackMe components. (splk-dsm, splk-dhm, splk-mhm)

• By default, only the splk-dsm component is enabled.

• If your environment is tiny enough (less than 2/3 TB per day of ingestion), you totally can leave the default configuration and create the tracker during this stage. (so leave “Yes” for Create Hybrid tracker)

• For larger environments, and up to any super large environments, we recommend to disable the creation of the Virtual Tenant and create the tracker later on manually via the Hybrid Tracker management UI.

• In the next screen, the splk-dhm component is disabled by default, we will leave disabled, this component is designed to track hosts and we will discuss it later on in this tutorial.

• In the next screen, the splk-mhm component is disabled by default, we will leave disabled, this component is designed to track metrics and we will discuss it later on in this tutorial.

• The final configuration screen and its top section allows defining TrackMe indexes, in most cases, the default indexes are fine, and we can leave the default configuration.

• The second part of that same screen allows defining ownership and permissions, all of these can be easily modified via TrackMe at point in time, in short:

Owner:

• Default owner is nobody, which refers to the splunk-system-user, this means that TrackMe will re-assign automatically the ownership of the knowledge objects to the splunk-system-user.

• This is fine for most cases and is a common practice in Splunk.

• On large or strict environments, creating a service account can be considered as good practices and notably allows tracking easily TrackMe’s related activity, leveraging Splunk Workload Management, see: :ref: trackme_admin_config for more information.

Permissions:

• The default permissions leverages 3 dimensions models, from admin to power and finally read only users.

• As a start, you generally can leave everything as a default

Create an hybrid tracker for splk-dsm

We will now create an hybrid tracker for the splk-dsm component, enter the freshly create Virtual Tenant:

• Scroll down to the scope of the tracker root constraint:

  • The default root constraint tracks all indexers, and excludes summary data (stash) and TrackMe related data.

  • On large environments, the scaling concept for TrackMe is to dedicate Hybrid Trackers to specific indexes scope, so trackers can share the load and be executed concurrently, allowing TrackMe to scale at any level.

  • On relatively small environments, the default configuration for a single tracker is in most cases able to cope with the load with no issues.

  • Finally, if you are a free community edition user, you can create up to two Hybrid Trackers by component.

  • For more information about scaling TrackMe for high scale environments, please refer to the TrackMe documentation, notably: Large Scale Environment and Best Practices Configuration Guide

About Trackers scope:

• When creating a Hybrid Tracker, TrackMe creates a dedicated Splunk macro which contains the scope of the tracker.

• You can therefore easily start with a restricted scope such as a list of index patterns, then modify this macro at point in time to include additional contexts.

• To retrieve the macro name after the tracker is created, click on Manage Hybrid tracker from the home UI.

• Scroll down to the break by logic:

  • By default, the behaviour is to create TrackMe entities as the combination of indexes and sourcetypes, therefore: <index>:<sourcetype>, in most cases, this is what you want.

  • An alternative, and depending on your requirements, is to use the merge mode, in this case TrackMe will create one entity per index: <index>.

  • Finally, you can also specify a custom list (comma separated format) of fields as part of a custom break by, this is more sophisticated and is generally used for specific use cases such as leveraging custom indexed fields in large scale environments.

• Time window definition:

  • By default, TrackMe uses a -4h/+4h time range for the search logic, this is generally fine for most cases.

  • In some contexts such as very high scale contexts, or limited capacity contexts, you can eventually reduce the earliest to reduce the associated costs and run time. (most likely for splk-dhm than splk-dsm)

• Benchmarking the tracker:

  • TrackMe allows to benchmark the most expensive part of the search logic, this provides an accurate enough idea of how long is going to take the tracker to execute.

  • The target is to execute the search every 5 minutes. (while this is under your control, in most cases we want to get a close as possible to this frequency, especially for feeds tracking)

  • This means that the search needs to be executed in less than 300 seconds to avoid generating skipping searches, and missing executions.

  • The scaling idea underneath is to create multiple trackers with a reduced scope, so they can be executed concurrently, and therefore reduce the execution time.

• Continue the process, until the tracker is created.

• You can request its immediate execution:

• After the tracker is executed, TrackMe will show up with entities created, and we can then continue our initial setup!

Step 2: Let’s create a Virtual Tenant for data host sources

We will now create a Virtual Tenant to start tracking hosts, we recommend to setup hosts tracking in a dedicated tenant, so we can easily manage various features such as disabling ML Outliers:

• component: splk-dhm

• tenant name: endpoints

• Continue up to the creation of the Virtual Tenant, enter the new tenant and access the Hybrid tracker wizard:

• We recommend being very strict with splk-dhm, notably we recommend restricting the scope as much as possible, and ensuring to allow only indexes that related to data generated by endpoints themselves.

• You totally can start with a certain scope, so you can first monitor the costs of the associated backends in TrackMe, then gradually increase the scope as needed.

• Finally you can also choose to include or not splunkd itself, if you wish to monitor the availability of Splunk UFs/UFs internals, which reflect a healthy connectivity and status of Splunk agents.

Create an hybrid tracker for splk-dhm

• You can increase the time span value up to 5m to reduce the associated computing costs, by increasing this value we reduce the strict accuracy of the latency calculation but this doesn’t affect the delay calculation.

• Very often when tracking hosts, you essentially care about detecting hosts that have stopped emitting data for particular conditions, increasing this value reduces costs without altering this capability at all.

• After the tracker is executed, TrackMe will show up with entities created, and we can then continue our initial setup!

Step 3: Alerting and notifications

At this stage, we have a valid deployment which monitors and tracks for various types of anomalies, from delay or latency to eventually ML anomalies, we can now setup alerting and notifications:

• In every Virtual Tenant, we want to create a technical alert which is going to leverage the builtin Alert actions in TrackMe, such as the automated Acknowledgements and Notable events creations.

• In addition, we will create a Notable alert which looks at notable events created by TrackMe, and optionally sends notifications to third parties. (emails, ITSM incident creation, etc)

• Refer to Alerting Architecture & Third Parties Integration for more insights about alerting and notifications in TrackMe.

Proceed as follows, and repeat in each tenant:

Technical Alert to handle TrackMe alert actions

• Open the Virtual Tenants and access to “TRACKING ALERTS” tab:

• Click on “Create a new Alert” and proceed with the alert creation wizard:

Notable Alert

• In the same screen, the first tab allows creating a new Notable alert from TrackMe:

• Modifying an alert is very simple, you can access to the alert modification screen shortcut as follows:

• For instance, say we have created an Notable alert for high/critical priority entities:

TrackMe alert for a Splunk high/critical priority entity:

- tenant: $result.tenant_id$
- object: $result.object$
- object_category: $result.object_category$
- state: $result.state$
- priority: $result.priority$
- anomaly_reason: $result.anomaly_reason$
- status_message: $result.status_message$
- drilldown_link: $result.drilldown_link$

Adapt to your preferences, repeat as needed, TrackMe is now Production ready!