Operational status Virtual Tenants

What is the Ops status for Virtual Tenants

TrackMe Virtual Tenants are themselves continuously monitored to report any kind of failures that would be encountered by any of the application trackers.

TrackMe trackers can fail when technical issues are encountered, this can happen for various reasons, such as:

  • Overloaded environments, where there are not search slots available (although TrackMe has a concept of automated re-attempts)

  • Search Head is in detention (overused file-systems, etc)

  • Network connectivity issues, credentials issues

  • Corrupted configuration (ex: missing Knowledge Objects or incorrect permissions)

  • And many more

With the goal of providing the highest visibility and quality, every Tracker and related custom commands carefully report failures to a central component called the components register.

In addition, when a Virtual Tenant is created, a Health Tracker is created which investigates the component register results, reports and generates Health events continuously.

You can easily observe the operation status of TrackMe Virtual Tenants in the Virtual Tenants user interface:

screen1.png

Click on the chart to open the detailed view:

screen2.png

Click on “Open in search” to access detailed per tracker results in the Splunk Search user interface:

screen3.png

Health events are stored in the audit indexe(s) of the Virtual Tenant(s), you can search for Health failures events easily, such as:

index=trackme_audit sourcetype="trackme:health" last_status="failure"
| timechart count by tenant_id
screen4.png

Example of failure detection

In this example, a remote Splunk Deployment is encountering issues due to a file-system being filled over the accepted limit by Splunk, therefore this instance does not accept searches any longer.

We can observe the Vetants Ops status reporting issues currently, as well as which Virtual Tenants are affected currently:

screen5.png

Accessing the detailed Ops status view quickly shows the affected tenants, related accounts and technical root cause:

screen6.png

We can access the live status from REST:

screen7.png

Additional resources

An additional (UFS) dashboard is available at:

  • Navigation Bar / Audit & Troubleshoot / Audit - Operational Statuses

screen8.png

Virtual Tenants Ops status technical overview

When a Virtual Tenant is created, a Health tracker is created and executed every 5 minutes:

trackme_health_tracker_tenant_<tenant_id>

This tracker executes the following custom command:

| trackmetrackerhealth tenant_id="<tenant_id>"

This custom command performs various operations and relies on the REST endpoint:

screen9.png screen10.png 
| trackme mode=post url="/services/trackme/v2/configuration/get_tenant_ops_status"

TrackMe components, such as Hybrid Trackers and their related custom commands call the components register at the backend level, and these states are stored in a Python dictionanry in the central store KVstore collection:

| inputlookup trackme_virtual_tenants | eval keyid=_key
| fields tenant_id, tenant_objects_exec_summary
screen11.png