TrackMe Sourcetypes & Metrics

About TrackMe sourcetypes and metrics

  • TrackMe generates various events and metrics for the purposes of its activity.

  • The destination indexes for both events and metrics are entirely configurable, and potentially on a per Virtual Tenant basis.

  • Events generated by TrackMe are JSON events, and parsed using KV_MODE=json.

  • Metrics are generated and stored in the Splunk metrics store using metric indexes as per the Virtual Tenant configuration.

TrackMe events & sourcetypes

The following event sourcetypes are generated by TrackMe:

To list all events from a Splunk search:

| tstats count where index=trackme_* by sourcetype

Sourcetype

Purpose

trackme:audit

TrackMe audit events, for instance when modifications are made against TrackMe entities

trackme:flip

TrackMe flipping state events, flipping events are generated when entities switch from one state to another

trackme:handler

Handler events are generated when TrackMe logics, such as trackers, are executed against a given entity. These events allow identifying which logics are maintaining and performing operations on which entities.

trackme:health

Health events are generated to monitor the health status of TrackMe components and logics, for instance these events are used to identify when a given Virtual Tenant is degraded due to an issue in a tracker.

trackme:sla_breaches

TrackMe has a concept of SLA tracking, when entities are breaching their SLA and the associated SLA class definition, an SLA breach event will be generated. (one event every 24 hours by default)

trackme:state

State events are generated when entities are maintained by TrackMe, and contain key information about the entities statuses.

trackme:stateful_alerts

Stateful events are generated via the concept of state aware alerting, when incidents are opened, updated or closed.

trackme:notable

TrackMe notable events produced by the TrackMe notable alert action

trackme:smart_status

TrackMe SmartStatus events produced by the TrackMe SmartStatus alert action, when an alert triggers and has the SmartStatus alert action enabled, automated investigations are performed and their results are indexed as SmartStatus events.

trackme:wlk:version_id

These events are specific to the Workload component (splk-wlk), and are generated when an updated version (or first time discovered) of a monitored Splunk scheduled search occurs.

TrackMe main metrics

TrackMe generates various metrics per component, using the following strict convention:

To list all metrics from a Splunk search:

| mcatalog values(metric_name) as metrics, values(_dims) as dimensions where index=trackme_metrics

To view the content of metrics in a practical way such as you would with events in Splunk:

| mpreview index=trackme_metrics filter="metric_name=trackme.*"

metric_name

Purpose

trackme.splk.feeds.*

Metrics generated by the Feeds tracking components (splk-dsm/splk-dhm/splk-mhm), which contain notably metrics are latency and availability for TrackMe feeds entities.

trackme.splk.flx.*

Metrics generated by the Flex Object splk-flx component. In Flex, a tracker can generate any kind of metrics depending on the use cases, from system metrics to functional or business metrics.

trackme.splk.cim.*

Metrics generated by the CIM compliance component. (splk-cim)

trackme.splk.wlk.*

Metrics generated by the Workload component. (splk-wlk)

trackme.components_register.*

Metrics generated when trackers are executed, which trace the health status and runtime of the various TrackMe components and logics such as trackers.

trackme.sla.object_state

These metrics are generated for SLA tracking purposes, and contain the state of the entities in a numerical format.

TrackMe logging events

About TrackMe logging

  • TrackMe has essentially two types of components generating log messages, REST API endpoints and custom commands.

  • Both carefully perform message logging, according to the logging level defined in the TrackMe global configurations.

REST API endpoints log messages

To access REST API log messages, you would use the following search:

index=_internal sourcetype=trackme:rest_api

Custom commands log messages

To access custom commands log messages, you would use the following search:

index=_internal sourcetype=trackme:custom_commands:*