Alerting Architecture & Third Parties Integration

Introduction to Alerting in TrackMe

Hint

Create Notable & SLA alerts directly from TrackMe since the release 2.0.91

• Since the release 2.0.91, you can create TrackMe Notable alerts, as well as SLA alerts directly from the TrackMe user interface.

• The present documentation details the concepts and architecture of TrackMe alerts, and how to integrate third parties alerting notifications, and remain valid regarding the builtin creation in TrackMe.

• However, from this version, please use TrackMe to create Notable alerts which allows TrackMe to maintain their orchestration and knowledge. (such as re-assignment, deletion, etc)

TrackMe alerts are designed to leverage the detection of issues affecting one or more entities for a TrackMe component:

• TrackMe Alerts are Splunk alerts created and maintained in TrackMe, via the builtin user interface

• Alerts are specific per Virtual Tenant and per TrackMe component

• TrackMe manages alerts knowledge for the purposes of maintaining the state of a Virtual Tenant, such as managing its enablement and deletion

• TrackMe alerts include different features with builtin alert actions, such as the TrackMe Notable action to generate and index a notable event when the alert triggers, which provides valuable context for the investigation purposes

• When creating scheduled knowledge objects, TrackMe automatically attempts to randomise the cron schedule for optimization purposes

Hint

Sending TrackMe alerts to third parties such as sending emails, channel messages or opening incidents in your ITSM

• You can easily send a notification to a third party via Splunk natively supported alert actions such as emails, or via Splunk Add-ons

• For instance, you may want to create a ticket in your ITSM software such as ServiceNow or Jira when the TrackMe alert triggers

• Another typical example can be to create a message in a Slack or Microsoft Teams channel

• The best design to have TrackMe alerts triggering its own actions, and rely on TrackMe Notable events to create incidents in your ITSM as needed for instance, the present documentation covers both designs in details

drilldown_link and Trackme UI integration

• Since TrackMe 2.0.88, you can submit params in the Home URL to filter on a tenant, component and entity, and automatically open the entity overview screen.

• A submitted URL will look like this: https://splunk.mydomain.com/en-GB/app/trackme/trackMe_home?tenant_id=mytenant&component=dsm&alias=myentity

• TrackMe notable events will also include a drilldown_link field, you can define the Splunk root URL at the time of the alert creation, or later on by editing the alert in Splunk.

• Notable events, the TrackMe alert action auttomatically builds and encodes the URL for you, so you can easily access the entity in the TrackMe user interface.

• You can also build the URL yourself as in the example above, notably if you do alert based on Notable events.

Creating an alert in TrackMe

You can create a new alert in the scope of any Virtual Tenant:

Available option may differ per TrackMe component, for instance with splk-dsm:

Once created, the alert reports in the alert user interface, and can be managed from this screen:

TrackMe modular alert actions

TrackMe has several alert actions that are leveraged when creating alerts:

• TrackMe Notable event: This alert action generates a JSON event in the Virtual Tenant notable index summarising the status of an entity when the alert triggers

• TrackMe Auto Acknowledgment: This alert action automatically acknowledges the entity in TrackMe’s workflow for a given period of time

• TrackMe SmartStatus: This alert performs automated investigations when an alert triggers for one or more entities, to improve and speed the root cause identifications and add context to an issue detected by TrackMe

The TrackMe Notable event is a mandatory alert action, others can be disabled during the creation of the alert if you wish to do so.

Alerts Triggering

When the alert fires and triggers results, the user interface shows a summary of the triggers and the actions involved:

The alert user interface allows to investigate alert actions traces:

From the entities perspective, we can for instance observe that the entity alert was acknowledged automatically:

A notable event was created:

As well as a TrackMe SmartStatus event:

Third Parties Alert Integration

Introduction to Third Parties Alerting Notification

Any TrackMe alert can be extended to send notifications to a third party, typically:

• Sending emails to a recipient, such as the owner of the entities/service, or TrackMe administrators and analysts

• Opening an incident to your ITSM such as ServiceNow or Atlassian Jira

• Creating a message in a Slack or Microsoft Teams channel

Note

Two main alerting architecture design can be implemented:

• Architecture scenario 1: Notable based design TrackMe alerts generate notable events, a centralised Splunk alert monitors for TrackMe notables and run alert actions for the third party

• Architecture scenario 2: Per alert design TrackMe alerts are configured to run the third party alert action and perform the associated action directly

While the two options are possible, the Architecture scenario 1 based on Notables has multiple advantages:

• Each TrackMe alert handles all entities, ensuring the creation of TrackMe notable events and other features such as the acknowledgment

• Allows centralising the definition of what should lead to the creation of an incident, sending an email or sharing channel message

• Notable events are normalised and provide all necessary information in a consistent fashion

• For instance, you can have a single “TrackMe forward notable” alert which restricts to high priority entities and any additional logic of your own, easily in SPL

• This logic can easily be maintained over time, without having to modify multiple alerts and provides a deeper and more consistent control

This documentation covers both approaches in the next sections.

Architecture scenario 1: TrackMe Notable Events

Introduction: Anatomy of TrackMe Notable Events

TrackMe Notable Events creation

TrackMe notable events are generated when a TrackMe alert triggers against one or more entities, relying on the built in TrackMe Notable modular alert action.

This alert action configuration is also visible when editing a TrackMe alert:

Note

Notable events are ONLY created when a TrackMe alert is configured and triggers, in this scenario all eligible entities should therefore be covered by an alert per tenant and component.

TrackMe Notable Events structure

TrackMe Notable events are JSON formatted events, these contain the following key information:

Field	Description
tenant_id	The tenant identifier this Notable event is related to.
object	The object name for this entity, this basically is the name of entities in TrackMe
object_category	A TrackMe identifier for each component, this describes which type of TrackMe component this notable is related to.
keyid	The unique identifier for this object in the tenant / component central KVstore collection
priority	The priority definition for this object (low / medium / high)
state	The status of the object (blue / green / orange / red)
anomaly_reason	A normalised field describing the reason behind the status of a given entity
status_message	A human readable detailed description of the reason why TrackMe generated an alert for this entity
timeStr	The time of the event in human readable format
drilldown_link	A link to the TrackMe Home user interface which filters on the tenant, component and entity, and automatically opens the entity overview screen (from TrackMe 2.0.88)
event_id	The unique identifier for this notable event, as the sha-256 hash of the notable event content (from TrackMe 2.1.0)
properties	The complete TrackMe object record in a nested JSON structure, as it was when the alert triggered

Therefore, TrackMe Notable events contain all necessary information in a consistent and normalised manner, to interact with any third party.

Example of a TrackMe Notable event:

The properties field contains the entire entity record as nested JSON, it is also extracted automatically so you can use fields as needed:

TrackMe Notable events are easy to access and review in TrackMe’s user interface:

Step 1: Create TrackMe alerts for tenants and components in used

The first step is to create a TrackMe alert for every tenant and components that are in use.

It is not necessary at this stage to filter on concepts like the priority because filtering will be made while monitoring the TrackMe notable events.

For each tenant, access the TRACKING ALERTS tab, and use the wizard to create an alert, you can leave all options to their default:

Step 2: Create a custom alert monitoring TrackMe notable

The second step is to create a custom Splunk alert which searches for TrackMe notable and enables third party interactions, for instance:

As a basis:

index=trackme_notable tenant_id=* priority=*
| table tenant_id, keyid, event_id, object, object_category, state, priority, anomaly_reason, status_message, drilldown_link

For instance, a typical use case would be to create incidents only for high priority entities in alert:

index=trackme_notable tenant_id=* priority="high"
| table tenant_id, keyid, event_id, object, object_category, state, priority, anomaly_reason, status_message, drilldown_link

Note

In TrackMe, you can define different notable indexes per tenant, if you did so you will want to adapt your search.

Create the Splunk alert:

• Cron every 5 minutes: */5 * * * *

• Earliest time: -15m

• Trigger for each result

• Throttle against the keyid for 60m

Then enable the third party alert action, for example we will enable creating incident in JIRA, the following shows an example of content for the alert which calls tokens replacements:

TrackMe alert for a Splunk high priority entity:

- tenant: $result.tenant_id$
- object: $result.object$
- object_category: $result.object_category$
- state: $result.state$
- priority: $result.priority$
- anomaly_reason: $result.anomaly_reason$
- status_message: $result.status_message$
- drilldown_link: $result.drilldown_link$

Hint

Re-assigning to a service account

• As a good practice, you should re-assign the created Splunk alert to the TrackMe service account user, or nobody

• You should avoid running alerts on behalf of normal users

Adding third parties actions:

Atlasian Jira example:

Note: this integration relies on the following Add-on for Jira: https://splunkbase.splunk.com/app/4958

Email example:

Note: Sending emails is a builtin alert action in Splunk

Slack example:

Note: this integration relies on the following Add-on for Slack: https://apps.splunk.com/app/2878

Optional: Notable events enrichment

When using Notable events as the central means for third party alerting, further logic can easily be implemented such as custom normalisation and enrichment.

With enrichment for instance, you may want to leverage your CMDB knowledge to add context information such as the owner or contact address email for the notification, there are plenty of options and designs possible.

TrackMe allows to lookup easily in your CMDB to provide context information handy in the user interface, you can leverage the same content from the central Notable alert, for more information about the CMDB Lookup integrator see: CMDB Lookup Integration

CMDB integration in the user interface:

For instance, we could update our Notable event alert:

index=trackme_notable tenant_id=* priority="high"
| lookup my_cmdb index as properties.data_index, sourcetype as properties.data_sourcetype OUTPUT
| table tenant_id, keyid, event_id, object, object_category, state, priority, anomaly_reason, status_message, drilldown_link

Enrichment information are automatically available:

Any of these fields can be used equally in your alert configuration using tokens, for instance we could update the recipient email address and add some logic to handle unknown entities:

index=trackme_notable tenant_id=* priority="high"
| lookup my_cmdb index as properties.data_index, sourcetype as properties.data_sourcetype OUTPUT
| eval contact=if(isnull(contact), "support@trackme-solutions.com", contact)
| table tenant_id, keyid, event_id, object, object_category, state, priority, anomaly_reason, status_message, drilldown_link, contact

And recycle this token instead:

Architecture scenario 2: Per TrackMe Alert Design

Another design choice is to configure third party alert actions on a per TrackMe alert instead of relying on the Notable events.

Note

This integration can also be considered as a valid choice, eventually easier and more logical, however it does not present some of the advantages in comparison with a TrackMe Notable based design:

• Each alert needs to be configured individually, which can be more challenging to maintain over time

• Entities that are not covered by any TrackMe alert will not lead to TrackMe alert actions being executed (Notable, Ack, SmartStatus), for instance if you restrict alerts using entities priority or tags and some entities are out of the scope

• Normalisation, enrichment or the introduction of custom SPL logic is decentralised opposed to the central alert based on Notables

• Finally, you may need more alerts to be created (for instance to cover entities by priority and have alert actions executed for all entities)

For the purposes of this documentation, the following scenario will be implemented:

• A first alert is created which targets high priority entities, when triggering a notification is sent using a Splunk alert action to open an incident in our ITSM tool (we will use Jira but this is applicable to any Splunk supported third party)

• A second alert is created which targets medium priority entities, this time an email will be sent when the alert triggers

• Entities set as low are not subject to a third party integration, however an alert should also be active for notable events and other TrackMe alert actions to be triggered

Step 1: Create a TrackMe alert for high priority entities

We start by creating an alert in TrackMe for high priority entities:

Repeat the same process for other priority levels, all entities should be covered by alerts, in this example we would therefore need 3 alerts in total.

Step 2: Enable alert actions per alert

The next step is to update TrackMe alerts to include the alert actions to be enabled:

This opens the Splunk alert edition management screen, we will add a new alert action:

We add for instance add the Jira alert action and define some of the most use fields returned by TrackMe:

These steps are the same as in the architecture scenario 1 Notable based, consult the previous section for more insights.

Context body example:

TrackMe alert for a Splunk high priority entity:

- tenant: $result.tenant_id$
- object: $result.object$
- object_category: $result.object_category$
- state: $result.state$
- priority: $result.priority$
- anomaly_reason: $result.anomaly_reason$
- status_message: $result.status_message$
- drilldown_link: $result.drilldown_link$

The same process needs to be achieved for every alert that should monitor TrackMe entities, per tenant and component.

When alerts trigger, Splunk automatically configured the alert actions according to your settings.

Deleting TrackMe alerts

Warning

TrackMe alerts are orchestrated by TrackMe and should be deleted in TrackMe

• When creating a TrackMe alert, it will preserve knowledge of it

• This allows automatically managing assignment and Roles Based Access Control settings, according to the Virtual Tenant setup

• Therefore, you must delete a TrackMe alert via TrackMe, and not directly via Splunk

Deleting an alert from the user interface

You can delete TrackMe alerts from the user interface, because TrackMe maintains the knowledge of objects associated with a given Virtual Tenant, alerts should be deleted through the user interface or the REST API:

Deleting an alert from the REST API

You can as well delete an alert using the REST API and the following endpoint:

| trackme mode=post url="/services/trackme/v2/alerting/admin/del_alert" body="{'tenant_id':'mytenant', 'alert_name': 'TrackMe alert tenant_id:mytenant - Alert custom on splk-dsm'}"