Logical groups

A logical group associates several related entities into an ensemble and judges them collectively. As long as enough of the group stays healthy, a member that would normally go red is instead shown blue — the anomaly is acknowledged but suppressed from alerting. It is the right tool for active/passive pairs, load-balanced pools, and anywhere a single failing member is expected rather than alarming.

Logical groups are available on every monitoring component except CIM — DSM, DHM, MHM, FLX, FQM, and WLK.

How logical groups work

A group defines a set of member entities and a minimum green percentage (object_group_min_green_percent). On every cycle TrackMe computes the group’s green percentage (healthy members ÷ total members):

• if the green percentage is at or above the minimum, a member that would be red is demoted to blue — the healthy group carries it, and no alert fires;

• if the green percentage falls below the minimum, protection is lifted and red members alert normally.

                    LOGICAL GROUP
              +--------------------------+
              |  "firewalls"             |
              |  min_green_percent: 50%  |
              +------------+-------------+
                           |
                   +-------+-------+
                   |               |
               +---+---+     +-----+---+
               | fw-01 |     |  fw-02  |
               | GREEN |     |  RED    |
               +-------+     +---------+

GREEN: 1/2 = 50%  >=  threshold (50%)   →  fw-02 shown BLUE (protected), no alert

─────────────────────────────────────────────────────────────────────────────

Now fw-01 also goes RED:   GREEN 0/2 = 0%  <  threshold (50%)
                           →  protection lifted, both RED, alerts fire

This prevents alert storms for expected partial failures — active/passive failovers, rolling restarts, single-node maintenance in a pool.

Note

• Protection only applies to red members, and only when the group has more than one member.

• If every member is red, there is no protection (green % = 0) — by design.

• For small groups the threshold matters: a two-member group needs a threshold of 50% (one of two healthy) to protect; at a higher threshold neither is protected once one fails.

Logical-group protection is one of the two sources of the blue state; the other is the disruption queue. When a member is protected, its anomaly reason includes in_logical_group and its status message names the group and its health.

Group record

Each group is stored in a per-tenant collection (kv_trackme_common_logical_group_tenant_<tenant_id>) and carries:

Field	Meaning
object_group_name	The group’s name.
object_group_members	Its member entities (multi-value).
object_group_min_green_percent	The minimum green percentage for the group to protect its members.
object_group_members_green / object_group_members_red	The current healthy / unhealthy member counts, maintained each cycle.
object_group_mtime	When the group was last modified.

Use cases

• Active/passive pairs — firewalls, load balancers, or databases in HA pairs where only one member sends data at a time.

• Rolling maintenance — server pools where one member at a time is taken offline for patching.

• Redundant data pipelines — several entities carrying the same feed, tracked individually but assessed as a group.

• Geographically distributed systems — entities across regions where partial failure is expected and tolerable.

Walkthrough: pairing two firewalls

A concrete lifecycle, with a pair of firewalls (pan-zur-ha41-a / pan-zur-ha41-b) in active/passive mode.

1. Identify the pair. In Splunk, find the hosts that form the pair — here they share an index, sourcetype, and naming convention.

2. Review them in TrackMe. In the DHM Tenant Home, both hosts are green and healthy.

3. Associate them into a group. Select both entities, click Bulk edit, choose Logical Group Association → Associate in New Group, and name the group. TrackMe sets the min_green_percent automatically from the member count (50% for a pair).

4. Verify in the Logical Groups manager. From the Actions menu choose Manage: logical groups. The manager lists the group, its members, and the minimum green percentage, which you can adjust with the slider.

5. See the protection in action. When one firewall stops sending (an active/passive failover) it would normally go red — but because its partner is healthy and the group is above threshold, TrackMe shows it blue. The entity overview and status message confirm it is protected by its logical group (anomaly reason in_logical_group).

Managing groups

There are three interactive ways to manage logical groups, plus the REST API and the auto-group command below.

• The Logical Groups manager (Actions → Manage: logical groups) — the central screen to view, search, add, and remove groups and adjust each group’s minimum green percentage.

• Bulk edit — select entities and associate into a new group, associate into an existing group, or remove their current association.

• Per entity — from an entity’s actions, view, set, change, or clear its group membership.

Automating with the REST API

Every action is available through the REST API (resource groups splk_logical_groups for reads and splk_logical_groups/write for writes); all endpoints are called with mode=post. The common ones:

# List all groups in a tenant
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_collection"
    body="{'tenant_id': 'mytenant'}"

# Get one group
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/logical_groups_get_grp"
    body="{'tenant_id': 'mytenant', 'object_group_name': 'group-emea'}"

# Create or update a group and its members
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_add_grp"
    body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer',
           'object_group_members': 'entity1,entity2', 'object_group_min_green_percent': '50'}"

# Delete a group and clear its members' associations
| trackme mode=post url="/services/trackme/v2/splk_logical_groups/write/logical_groups_del_grp"
    body="{'tenant_id': 'mytenant', 'object_group_name': 'group-amer'}"

Further endpoints exist for bulk and per-entity association — see the REST API reference.

Automating at scale with trackmeautogroup

In a large estate you do not want to build groups by hand. The custom command ``trackmeautogroup`` creates, updates, and prunes logical groups automatically from an SPL search you write — the practical way to keep groups in sync with reality.

It is a streaming command: it consumes the entities your upstream SPL produces, and each input row must carry two fields:

• object_group_name — the name of the group the entity belongs to;

• object_group_members — the group’s members, as a multi-value field.

From those rows it:

• creates the group if it does not exist;

• updates the membership if it has changed, and sets object_group_min_green_percent automatically from the member count (round(100 / members, 2) — 50% for two, 33.33% for three, …);

• prunes a group down to nothing once only one active member remains — controlled by purge_single_member_grp (True by default; set False to keep single-member groups).

It requires tenant_id.

Identifying members at scale

The real work is deciding which entities belong together. Common approaches:

• Naming convention — extract a shared pair/zone identifier from the entity name with a regular expression.

• CMDB enrichment — join a Splunk lookup (your CMDB) to derive the grouping key.

• Custom logic — any SPL that emits object_group_name and object_group_members.

Example — by naming convention:

| inputlookup trackme_dhm_tenant_hosts-monitoring where monitored_state="enabled"
| fields object, alias
``` extract the pair identifier from the alias ```
| rex field=alias "(?<grp_prefix>.*)-(?<grp_suffix>ha\d{1,})-"
| eval object_group_name = grp_prefix . ":" . grp_suffix
| where isnotnull(object_group_name) AND object_group_name!=""
``` group members and keep only real groups (>1 member) ```
| stats values(object) as object_group_members by object_group_name
| eventstats dc(object_group_members) as dcount_members by object_group_name
| where dcount_members>1
``` create / update / prune the groups ```
| trackmeautogroup tenant_id="hosts-monitoring" purge_single_member_grp=True

Example — by CMDB lookup:

| inputlookup trackme_dhm_tenant_mytenant where monitored_state="enabled"
| fields object
| lookup my_cmdb_lookup host AS object OUTPUT pair_id
| eval object_group_name = "pair:" . pair_id
| where isnotnull(object_group_name) AND object_group_name!=""
| stats values(object) as object_group_members by object_group_name
| eventstats dc(object_group_members) as dcount_members by object_group_name
| where dcount_members>1
| trackmeautogroup tenant_id="mytenant" purge_single_member_grp=True

Scheduling it

Once the SPL is right, save it as a scheduled report to keep groups current as your estate changes (daily is usually enough). The command logs its actions to index=_internal sourcetype="trackme:custom_commands:trackmeautogroup".

Important

This scheduled report is yours, not orchestrated by TrackMe — so if you later delete the tenant, remember to delete the report too.

See also

• Disruption queue — the other source of the blue state.

• The entity state machine — where group protection fits in the state machine.

• REST API reference — the REST API reference.

• Virtual Groups — virtual groups (cross-tenant views) are a different feature.