Logical groups (entities ensemble association)

About Logical Groups

The Logical Groups feature in TrackMe allows to associate multiple entities into an ensemble, for the purposes of influencing the status of each entities depending on the status of the group itself
For instance, you can associate two entities into a group with a 50% minimal green threshold, as long as one of the two remains compliant the group will be considered healthy and the entities will be in a blue state
The Logical Groups features is available for the component of the Splunk feeds tracking family, as well as Flex Objects (splk-flx)

Introduction to logical groups

In TrackMe and for the Feeds tracking components, you can associate multiple entities into a logical group.

Logical groups are stored in a dedicated KVstore collection per tenant:

trackme_common_logical_group_tenant_<tenant_id>

A logical group record is composed by:

object_group_name: The name of the logical group
object_group_members: The members of the group (multi-value field)
object_group_min_green_percent: The minimal percentage of entities in a healthy state for the group to be considered healthy
object_group_mtime: The epoch time recorded for the last modification of the group

A logical group influences the status of entities that are member of a group as following:

If an entity is not healthy (red, orange) but the logical group itself is healthy, the entity status will be blue instead (because the minimal green percentage of healthy entities in the group is respected)
If an entity is not healthy and the group itself does not meet the minimal percentage of healthy entities in the group, the entity status will be in a non healthy state (red, orange)

Use cases for logical groups

There are various use cases where logical groups can be beneficial:

Multiple entities represent the same underneath data pipeline, however these entities need to be considered independently for different reasons
Group multiple entities according to your need, and avoid alerting if only a single entity is not healthy

Creating, updating and deleting logical groups via the central Logical Group screen

Since TrackMe 2.0.83, in addition with the per entity screen and the bulkd edit (see the next sections), you can also create, update and delete logical groups via the central Logical Group screen available from the TrackMe main menu:

Creating logical groups and associating entities

Logical groups can be defined through TrackMe user interfaces, in SPL via the REST API endpoint and the trackme command, as well as via dedicated custom command utility trackmeautogroup.

Logical groups via the UI and bulk edit

You can associate entities into a new or existing logical group from the bulk edition feature:

select one or more entities to be associated:

you can create a new group or choose an existing group:

From the bulk edit screen you can therefore:

Associate selected entities into a new logical group
Associate selected entities into an existing logical group
Remove the association of these entities with their current logical group

Logical groups per entity via the modification screen

On a per entity basis, click on the the Modify button and access to the logical group setting screen:

From this screen, you can:

View the current logical group association, if any
Associate this entity into a new logical group
Associate this entity into an existing logical group
Remove the association of this entity from its current logical group

Logical groups management via the REST API and the trackme command

Consult the REST API reference for the full list of available endpoints and their respective usage:

The API resource groups are:

splk_logical_groups: for read only operations
splk_logical_groups/write: for write operations

For instance:

list all groups:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_collection" body="{'tenant_id': 'mytenant'}"

Get a specific group:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_get_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-emea'}"

Create a new group and associate entities, update the group if exists already:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_add_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer', 'object_group_members': 'eventgen-firewall:netscreen:firewall|key:region;company|amer;company003,eventgen-firewall:netscreen:firewall|key:region;company|amer;company004', 'object_group_min_green_percent': '50'}"

Delete a group and clear all association for its members:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_del_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer'}"

Auto logical group management via the command trackmeautogroup

Hint

The command trackmeautogroup was introduced in release 2.0.36

The custom command trackmeautogroup is a streaming custom command utility which can be leveraged to automatically perform group creation, association and management according to your needs.

This is a streaming command which means that it will retrieve entities to be managed from a an upstream SPL logic you will define, it behaves the following way:

The command requires two fields to be provided by your SPL logic:
- object_group_name: the name of the logical group
- object_group_members: the members of the logical group in a multi-value format
Based on these upstream results, the command verifies if a group exists already and will create the group if necessary
If the group exists already, but the members definition differs, it will update the group automatically and define the minimal percentage expected for the group (ex: 50% if two entities, 33.33% if 3 entities, etc)
It will remove the group automatically if there is only 1 active member left, this can be controlled by the argument purge_single_member_grp=True|False

Example of implementation

Based on our prior example, we have a list of common criterias which define what a group definition should be:

same index and sourcetype
same “region”

In our example, any entity from the same “company” should be part of a group which belongs to these criterias (index / sourcetype / region), we can leverage this simple SPL logic with a regular expression to extract the relevant association:

| inputlookup append=t trackme_dsm_tenant_mytenant where monitored_state="enabled"
| fields object

``` Extract from the object fields which will be used for the logical group definition ```
| rex field=object "^(?<index>[^\:]*)\:(?<sourcetype>[^\|]*)\|key\:[^\|]*\|(?<region>[^\;]*)\;(?<company>.*)"

Note:

Note that we filter on entities for which the monitored_state is enabled, as there wouldn’t be any interest in maintaining groups for entities that are not monitored

Next, we want to define the logical group name, form the association and filter on entities which match our criterias:

| inputlookup append=t trackme_dsm_tenant_mytenant where monitored_state="enabled"
| fields object

``` Extract from the object fields which will be used for the logical group definition ```
| rex field=object "^(?<index>[^\:]*)\:(?<sourcetype>[^\|]*)\|key\:[^\|]*\|(?<region>[^\;]*)\;(?<company>.*)"

``` Set the logical group ```
| eval object_group_name = index . ":" . sourcetype . ":" . region

``` filter and only group eligible entities ```
| where isnotnull(object_group_name) AND object_group_name!=""

``` group the members and calculate the object_group_min_green_percent ```
| stats values(object) as object_group_members by object_group_name

This is all that we need! We can now call the custom command which takes of everything else for us:

| inputlookup append=t trackme_dsm_tenant_mytenant where monitored_state="enabled"
| fields object

``` Extract from the object fields which will be used for the logical group definition ```
| rex field=object "^(?<index>[^\:]*)\:(?<sourcetype>[^\|]*)\|key\:[^\|]*\|(?<region>[^\;]*)\;(?<company>.*)"

``` Set the logical group ```
| eval object_group_name = index . ":" . sourcetype . ":" . region

``` filter and only group eligible entities ```
| where isnotnull(object_group_name) AND object_group_name!=""

``` group the members and calculate the object_group_min_green_percent ```
| stats values(object) as object_group_members by object_group_name

| trackmeautogroup tenant_id="mytenant" purge_single_member_grp=True

See the collection:

| inputlookup trackme_common_logical_group_tenant_mytenant

Next executions of this logic with take care of adding any new group of it has more than 1 entity, and remove any group that has only 1 active entity left, logging the output of performed actions

Example: we remove 1 of these 4 entities:*

The custom command logs into:

index=_internal sourcetype="trackme:custom_commands:trackmeautogroup"

All that is left now is to save logic into a report, and shedule its execution according to your preference.

Note that as of now, this scheduled logic will not be orchestrated by TrackMe, therefore if you later on remove this tenant, you need to manually remove this report too.