Logical groups (entities ensemble association)¶
About Logical Groups
The Logical Groups feature in TrackMe allows you to associate multiple entities into an ensemble, for the purposes of influencing the status of each entity depending on the status of the group itself
For instance, you can associate two firewalls into a group with a 50% minimal green threshold; as long as one of the two remains compliant, the group will be considered healthy and the non-compliant entity will be in a blue state rather than red
The Logical Groups feature is available for the components of the Splunk feeds tracking family, as well as Flex Objects (splk-flx)
How logical groups work¶
Logical groups let you define an ensemble of related entities and a minimal percentage of healthy members required for the group to be considered healthy. When the group is healthy, any member that would normally be in a red or orange state is instead shown as blue — meaning the anomaly is acknowledged but suppressed from alerting.
The following diagram illustrates the concept with a pair of firewalls operating in active/passive mode:
LOGICAL GROUP
+--------------------------+
| "firewalls" |
| min_green_percent: 50% |
+------------+-------------+
|
+-------+-------+
| |
+---+---+ +-----+---+
| fw-01 | | fw-02 |
| GREEN | | RED |
+-------+ +---------+
GREEN: 1/2 = 50% >= threshold (50%)
-----------------------------------------
+----------------------------------------+
| fw-01 * GREEN (healthy) |
| fw-02 * BLUE (red, protected) |
+----------------------------------------+
No alert fires - one firewall down is OK.
==========================================
Now fw-01 also goes RED:
+-------+ +---------+
| fw-01 | | fw-02 |
| RED | | RED |
+-------+ +---------+
GREEN: 0/2 = 0% < threshold (50%)
-----------------------------------------
+----------------------------------------+
| fw-01 * RED (alert!) |
| fw-02 * RED (alert!) |
+----------------------------------------+
Both down - alerts fire.
In summary:
When enough members remain healthy (above
object_group_min_green_percent), anomalous members get blue state instead of red — alerts are suppressedWhen too many members fail and the threshold is breached, protection is lifted and red entities trigger alerts normally
This prevents alert storms for expected partial failures such as active/passive failovers, rolling restarts, or single-node maintenance in a pool
Group record structure¶
Logical groups are stored in a dedicated KVstore collection per tenant:
trackme_common_logical_group_tenant_<tenant_id>
A logical group record is composed of:
object_group_name: The name of the logical groupobject_group_members: The members of the group (multi-value field)object_group_min_green_percent: The minimal percentage of entities in a healthy state for the group to be considered healthyobject_group_mtime: The epoch time recorded for the last modification of the group
Use cases for logical groups¶
There are various use cases where logical groups can be beneficial:
Active/Passive pairs: Firewalls, load balancers, or database servers working in HA pairs where only one member sends data at a time
Rolling maintenance: Server pools where one member at a time may be taken offline for patching
Redundant data pipelines: Multiple entities representing the same underlying data feed that need to be tracked independently but assessed as a group
Geographically distributed systems: Entities across regions where partial failure is expected and tolerable
Walkthrough: pairing firewalls with logical groups¶
The following walkthrough demonstrates the full lifecycle using a pair of firewalls (pan-zur-ha41-a and pan-zur-ha41-b) operating in active/passive mode.
Step 1 - Identify the pair¶
In Splunk, identify the hosts that form a pair. In this example, both firewalls share the same index, sourcetype, and naming convention:
Step 2 - Review entities in TrackMe¶
In the Tenant Home for Data Host Monitoring (DHM), locate the pair. Both hosts are currently green and healthy:
Step 3 - Select and associate into a logical group¶
Select both entities and click the Bulk edit button:
In the bulk edit dialog, choose Logical Group Association as the category, Associate in New Group as the action, and enter a group name (here pan-zur-ha41):
TrackMe automatically sets the min_green_percent based on the number of members (50% for a pair of two).
Step 4 - Verify the group in the Logical Groups manager¶
Open the Actions menu and select Manage: logical groups:
The Logical Groups management screen shows the group with its members and the min green percentage, which can be adjusted using the slider:
Step 5 - Observe the blue state protection in action¶
When one firewall stops sending data (for instance during an active/passive failover), it would normally go red. Because it belongs to a logical group and the other member is still healthy, TrackMe shows it as blue instead:
Opening the Entity Overview for the affected host confirms the blue state. The status message explains that the entity is protected by the logical group membership:
The detailed status message shows the anomaly reason includes in_logical_group, along with the group health percentage and member information:
Creating, updating and deleting logical groups via the central Logical Group screen¶
In addition to the bulk edit walkthrough above, you can manage all logical groups from the central management screen accessible via the Actions menu > Manage: logical groups:
From this screen you can:
View all existing logical groups, their members, and min green percentage
Search for groups by name or member
Adjust the min green percentage using the slider
Add new groups
Remove selected groups
Logical groups via the UI and bulk edit¶
You can associate entities into a new or existing logical group from the bulk edition feature:
Select one or more entities to be associated:
Create a new group or choose an existing group:
From the bulk edit screen you can:
Associate selected entities into a new logical group
Associate selected entities into an existing logical group
Remove the association of these entities with their current logical group
Logical groups per entity via the modification screen¶
On a per-entity basis, click on the entity actions and access the logical group setting:
From this screen, you can:
View the current logical group association, if any
Remove the entity from its current logical group
Associate this entity to an existing logical group
Create a new logical group and associate this entity as a member
View all existing logical groups (read-only overview)
Logical groups management via the REST API and the trackme command¶
Consult the REST API reference for the full list of available endpoints and their respective usage.
The API resource groups are:
splk_logical_groups: for read-only operations
splk_logical_groups/write: for write operations
For instance:
List all groups:
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_collection" body="{'tenant_id': 'mytenant'}"
Get a specific group:
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_get_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-emea'}"
Create a new group and associate entities, update the group if it already exists:
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_add_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer', 'object_group_members': 'entity1,entity2', 'object_group_min_green_percent': '50'}"
Delete a group and clear all associations for its members:
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_del_grp" body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer'}"
Auto logical group management via the command trackmeautogroup¶
The custom command trackmeautogroup is a streaming custom command utility which can be leveraged to automatically perform group creation, association and management according to your needs.
This is a streaming command which means that it will retrieve entities to be managed from an upstream SPL logic you will define, it behaves in the following way:
The command requires two fields to be provided by your SPL logic:
object_group_name: the name of the logical groupobject_group_members: the members of the logical group in a multi-value format
Based on these upstream results, the command verifies if a group already exists and will create the group if necessary
If the group already exists, but the members definition differs, it will update the group automatically and define the minimal percentage expected for the group (ex: 50% if two entities, 33.33% if 3 entities, etc)
It will remove the group automatically if there is only 1 active member left, this can be controlled by the argument
purge_single_member_grp=True|False
Identifying pair members at scale¶
Setting up logical groups at scale requires identifying which entities belong together. This is generally the key challenge and there are several approaches:
Naming convention: If your hosts follow a predictable naming pattern containing a zone or pair identifier (e.g.,
pan-zur-ha41-a/pan-zur-ha41-b), you can extract the common prefix with a regular expressionCMDB enrichment: More realistically, you can rely on a CMDB under the form of a Splunk lookup file, and by enrichment you would define the association
Custom logic: Any SPL logic that produces the required
object_group_nameandobject_group_membersfields
Example using naming convention¶
When host names contain a predictable pair identifier, you can extract it directly with a regular expression:
| inputlookup append=t trackme_dhm_tenant_hosts-monitoring where monitored_state="enabled"
| fields object, alias
``` Extract the pair identifier from the alias ```
| rex field=alias "(?<object_group_prefix>.*)-(?<object_group_suffix>ha\d{1,})-"
| eval object_group_name= object_group_prefix . ":" . object_group_suffix
``` filter and only group eligible entities ```
| where isnotnull(object_group_name) AND object_group_name!=""
``` group the members and calculate the object_group_min_green_percent ```
| stats values(object) as object_group_members by object_group_name
| eventstats dc(object_group_members) as dcount_members by object_group_name
| where dcount_members>1
``` process the autogroup ```
| trackmeautogroup tenant_id="hosts-monitoring" purge_single_member_grp=True
Example using CMDB lookup enrichment¶
If your naming convention does not allow direct extraction, you can rely on a CMDB lookup for enrichment:
| inputlookup append=t trackme_dhm_tenant_mytenant where monitored_state="enabled"
| fields object
``` Enrich with CMDB lookup to get the pair identifier ```
| lookup my_cmdb_lookup host AS object OUTPUT pair_id
``` Set the logical group name ```
| eval object_group_name = "pair:" . pair_id
``` filter and only group eligible entities ```
| where isnotnull(object_group_name) AND object_group_name!=""
``` group the members ```
| stats values(object) as object_group_members by object_group_name
| eventstats dc(object_group_members) as dcount_members by object_group_name
| where dcount_members>1
``` process the autogroup ```
| trackmeautogroup tenant_id="mytenant" purge_single_member_grp=True
Scheduling the auto-group logic¶
Once your SPL logic is validated, save it as a scheduled report in Splunk to maintain groups automatically.
The command logs its actions into:
index=_internal sourcetype="trackme:custom_commands:trackmeautogroup"Subsequent executions will add new groups, update changed memberships, and remove groups that have only 1 active member left
Schedule frequency depends on how often your environment changes (daily is usually sufficient)
Note that as of now, this scheduled logic will not be orchestrated by TrackMe, therefore if you later on remove this tenant, you need to manually remove this report too.