Operational status Virtual Tenants

What is the Ops status for Virtual Tenants

Hint

TrackMe Virtual Tenants are themselves continuously monitored to report any kind of failures that would be encountered by any of the application trackers.

TrackMe trackers can fail when technical issues are encountered, this can happen for various reasons, such as:

  • Overloaded environments, where there are not search slots available (although TrackMe has a concept of automated re-attempts)

  • Search Head is in detention (overused file-systems, etc)

  • Network connectivity issues, credentials issues

  • Corrupted configuration (ex: missing Knowledge Objects or incorrect permissions)

  • And many more

With the goal of providing the highest visibility and quality, every Tracker and related custom commands carefully report failures to a central component called the components register.

In addition, when a Virtual Tenant is created, a Health Tracker is created which investigates the component register results, reports and generates Health events continuously.

You can easily observe the operation status of TrackMe Virtual Tenants in the Virtual Tenants user interface:

screen1.png

Click on the chart to open the detailed view:

screen2.png

Click on “Open in search” to access detailed per tracker results in the Splunk Search user interface:

screen3.png

Health events are stored in the audit indexe(s) of the Virtual Tenant(s), you can search for Health failures events easily, such as:

index=trackme_audit sourcetype="trackme:health" last_status="failure"
| timechart count by tenant_id
screen4.png

Example of failure detection

In this example, a remote Splunk Deployment is encountering issues due to a file-system being filled over the accepted limit by Splunk, therefore this instance does not accept searches any longer.

We can observe the Vetants Ops status reporting issues currently, as well as which Virtual Tenants are affected currently:

screen5.png

Accessing the detailed Ops status view quickly shows the affected tenants, related accounts and technical root cause:

screen6.png

We can access the live status from REST:

screen7.png

Clearing the Virtual Tenants Operational Status

From TrackMe 2.0.97, you can clear the Virtual Tenants Operational Status via the Virtual Tenants UI and from the REST API:

Access to the screen TrackMe Tenants Operational health statuses:

screen_clear_status.png

You can then request to clear for all tenants, or a selection of tenants and you can also choose to execute or not the health tracker during the clear process:

screen_clear_status2.png

Underneath, the UI calls the REST endpoint to clear the Virtual Tenants Operational Status:

screen_clear_status3.png

Example:

| trackme url="/services/trackme/v2/vtenants/admin/clear_exec_summary" mode="post" body="{'tenants': 'all'}"

Additional resources

An additional (UFS) dashboard is available at:

  • Navigation Bar / Audit & Troubleshoot / Audit - Operational Statuses

screen8.png

Virtual Tenants Ops status technical overview

When a Virtual Tenant is created, a Health tracker is created and executed every 5 minutes:

trackme_health_tracker_tenant_<tenant_id>

This tracker executes the following custom command:

| trackmetrackerhealth tenant_id="<tenant_id>"

This custom command performs various operations and relies on the REST endpoint:

screen9.png screen10.png 
| trackme mode=post url="/services/trackme/v2/configuration/get_tenant_ops_status"

TrackMe components, such as Hybrid Trackers and their related custom commands call the components register at the backend level, and these states are stored in a Python dictionanry in the central store KVstore collection:

| inputlookup trackme_virtual_tenants | eval keyid=_key
| fields tenant_id, tenant_objects_exec_summary
screen11.png