Logical groups (entities ensemble association)

About Logical Groups

The Logical Groups feature in TrackMe allows you to associate multiple entities into an ensemble, for the purposes of influencing the status of each entity depending on the status of the group itself
For instance, you can associate two entities into a group with a 50% minimal green threshold; as long as one of the two remains compliant, the group will be considered healthy and the entities will be in a blue state
The Logical Groups feature is available for the components of the Splunk feeds tracking family, as well as Flex Objects (splk-flx)

Introduction to logical groups

In TrackMe and for the Feeds tracking components, you can associate multiple entities into a logical group.

Logical groups are stored in a dedicated KVstore collection per tenant:

trackme_common_logical_group_tenant_<tenant_id>

A logical group record is composed of:

object_group_name: The name of the logical group
object_group_members: The members of the group (multi-value field)
object_group_min_green_percent: The minimal percentage of entities in a healthy state for the group to be considered healthy
object_group_mtime: The epoch time recorded for the last modification of the group

A logical group influences the status of entities that are members of a group as follows:

If an entity is not healthy (red, orange) but the logical group itself is healthy, the entity status will be blue instead (because the minimal green percentage of healthy entities in the group is respected)
If an entity is not healthy and the group itself does not meet the minimal percentage of healthy entities in the group, the entity status will be in a non-healthy state (red, orange)

Use cases for logical groups

There are various use cases where logical groups can be beneficial:

Multiple entities represent the same underlying data pipeline, however these entities need to be considered independently for different reasons
Group multiple entities according to your needs, and avoid alerting if only a single entity is not healthy

Creating, updating and deleting logical groups via the central Logical Group screen

Since TrackMe 2.0.83, in addition to the per-entity screen and the bulk edit (see the next sections), you can also create, update and delete logical groups via the central Logical Group screen available from the TrackMe main menu:

Creating logical groups and associating entities

Logical groups can be defined through TrackMe user interfaces, in SPL via the REST API endpoint and the trackme command, as well as via dedicated custom command utility trackmeautogroup.

Logical groups via the UI and bulk edit

You can associate entities into a new or existing logical group from the bulk edition feature:

select one or more entities to be associated:

you can create a new group or choose an existing group:

From the bulk edit screen you can:

Associate selected entities into a new logical group
Associate selected entities into an existing logical group
Remove the association of these entities with their current logical group

Logical groups per entity via the modification screen

On a per-entity basis, click on the Modify button and access the logical group setting screen:

From this screen, you can:

View the current logical group association, if any
Associate this entity into a new logical group
Associate this entity into an existing logical group
Remove the association of this entity from its current logical group

Logical groups management via the REST API and the trackme command

Consult the REST API reference for the full list of available endpoints and their respective usage:

The API resource groups are:

splk_logical_groups: for read-only operations
splk_logical_groups/write: for write operations

For instance:

list all groups:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_collection" body="{'tenant_id': 'mytenant'}"

Get a specific group:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_get_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-emea'}"

Create a new group and associate entities, update the group if it already exists:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_add_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer', 'object_group_members': 'eventgen-firewall:netscreen:firewall|key:region;company|amer;company003,eventgen-firewall:netscreen:firewall|key:region;company|amer;company004', 'object_group_min_green_percent': '50'}"

Delete a group and clear all associations for its members:

| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_del_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer'}"

Auto logical group management via the command trackmeautogroup

Hint

The command trackmeautogroup was introduced in release 2.0.36

The custom command trackmeautogroup is a streaming custom command utility which can be leveraged to automatically perform group creation, association and management according to your needs.

This is a streaming command which means that it will retrieve entities to be managed from an upstream SPL logic you will define, it behaves in the following way:

The command requires two fields to be provided by your SPL logic:
- object_group_name: the name of the logical group
- object_group_members: the members of the logical group in a multi-value format
Based on these upstream results, the command verifies if a group already exists and will create the group if necessary
If the group already exists, but the members definition differs, it will update the group automatically and define the minimal percentage expected for the group (ex: 50% if two entities, 33.33% if 3 entities, etc)
It will remove the group automatically if there is only 1 active member left, this can be controlled by the argument purge_single_member_grp=True|False

Example of implementation

Based on our prior example, we have a list of common criteria which define what a group definition should be:

same index and sourcetype
same “region”

In our example, any entity from the same “company” should be part of a group which belongs to these criteria (index / sourcetype / region), we can leverage this simple SPL logic with a regular expression to extract the relevant association:

| inputlookup append=t trackme_dsm_tenant_mytenant where monitored_state="enabled"
| fields object

``` Extract from the object fields which will be used for the logical group definition ```
| rex field=object "^(?<index>[^\:]*)\:(?<sourcetype>[^\|]*)\|key\:[^\|]*\|(?<region>[^\;]*)\;(?<company>.*)"

Note:

Note that we filter on entities for which the monitored_state is enabled, as there wouldn’t be any interest in maintaining groups for entities that are not monitored

Next, we want to define the logical group name, form the association and filter on entities which match our criteria:

| inputlookup append=t trackme_dsm_tenant_mytenant where monitored_state="enabled"
| fields object

``` Extract from the object fields which will be used for the logical group definition ```
| rex field=object "^(?<index>[^\:]*)\:(?<sourcetype>[^\|]*)\|key\:[^\|]*\|(?<region>[^\;]*)\;(?<company>.*)"

``` Set the logical group ```
| eval object_group_name = index . ":" . sourcetype . ":" . region

``` filter and only group eligible entities ```
| where isnotnull(object_group_name) AND object_group_name!=""

``` group the members and calculate the object_group_min_green_percent ```
| stats values(object) as object_group_members by object_group_name

This is all that we need! We can now call the custom command which takes of everything else for us:

| inputlookup append=t trackme_dsm_tenant_mytenant where monitored_state="enabled"
| fields object

``` Extract from the object fields which will be used for the logical group definition ```
| rex field=object "^(?<index>[^\:]*)\:(?<sourcetype>[^\|]*)\|key\:[^\|]*\|(?<region>[^\;]*)\;(?<company>.*)"

``` Set the logical group ```
| eval object_group_name = index . ":" . sourcetype . ":" . region

``` filter and only group eligible entities ```
| where isnotnull(object_group_name) AND object_group_name!=""

``` group the members and calculate the object_group_min_green_percent ```
| stats values(object) as object_group_members by object_group_name

| trackmeautogroup tenant_id="mytenant" purge_single_member_grp=True

See the collection:

| inputlookup trackme_common_logical_group_tenant_mytenant

Next executions of this logic with take care of adding any new group of it has more than 1 entity, and remove any group that has only 1 active entity left, logging the output of performed actions

Example: we remove 1 of these 4 entities:*

The custom command logs into:

index=_internal sourcetype="trackme:custom_commands:trackmeautogroup"

All that is left now is to save logic into a report, and shedule its execution according to your preference.

Note that as of now, this scheduled logic will not be orchestrated by TrackMe, therefore if you later on remove this tenant, you need to manually remove this report too.