Operational status Virtual Tenants
What is the Ops status for Virtual Tenants
Hint
TrackMe Virtual Tenants are themselves continuously monitored to report any kind of failures that would be encountered by any of the application trackers.
TrackMe trackers can fail when technical issues are encountered, this can happen for various reasons, such as:
Overloaded environments, where there are not search slots available (although TrackMe has a concept of automated re-attempts)
Search Head is in detention (overused file-systems, etc)
Network connectivity issues, credentials issues
Corrupted configuration (ex: missing Knowledge Objects or incorrect permissions)
And many more
With the goal of providing the highest visibility and quality, every Tracker and related custom commands carefully report failures to a central component called the components register.
In addition, when a Virtual Tenant is created, a Health Tracker is created which investigates the component register results, reports and generates Health events continuously.
You can easily observe the operation status of TrackMe Virtual Tenants in the Virtual Tenants user interface:
![screen1.png](_images/screen137.png)
Click on the chart to open the detailed view:
![screen2.png](_images/screen226.png)
Click on “Open in search” to access detailed per tracker results in the Splunk Search user interface:
![screen3.png](_images/screen313.png)
Health events are stored in the audit indexe(s) of the Virtual Tenant(s), you can search for Health failures events easily, such as:
index=trackme_audit sourcetype="trackme:health" last_status="failure"
| timechart count by tenant_id
![screen4.png](_images/screen412.png)
Example of failure detection
In this example, a remote Splunk Deployment is encountering issues due to a file-system being filled over the accepted limit by Splunk, therefore this instance does not accept searches any longer.
We can observe the Vetants Ops status reporting issues currently, as well as which Virtual Tenants are affected currently:
![screen5.png](_images/screen511.png)
Accessing the detailed Ops status view quickly shows the affected tenants, related accounts and technical root cause:
![screen6.png](_images/screen68.png)
We can access the live status from REST:
![screen7.png](_images/screen79.png)
Clearing the Virtual Tenants Operational Status
From TrackMe 2.0.97, you can clear the Virtual Tenants Operational Status via the Virtual Tenants UI and from the REST API:
Access to the screen TrackMe Tenants Operational health statuses:
![screen_clear_status.png](_images/screen_clear_status.png)
You can then request to clear for all tenants, or a selection of tenants and you can also choose to execute or not the health tracker during the clear process:
![screen_clear_status2.png](_images/screen_clear_status2.png)
Underneath, the UI calls the REST endpoint to clear the Virtual Tenants Operational Status:
![screen_clear_status3.png](_images/screen_clear_status3.png)
Example:
| trackme url="/services/trackme/v2/vtenants/admin/clear_exec_summary" mode="post" body="{'tenants': 'all'}"
Additional resources
An additional (UFS) dashboard is available at:
Navigation Bar / Audit & Troubleshoot / Audit - Operational Statuses
![screen8.png](_images/screen88.png)
Virtual Tenants Ops status technical overview
When a Virtual Tenant is created, a Health tracker is created and executed every 5 minutes:
trackme_health_tracker_tenant_<tenant_id>
This tracker executes the following custom command:
| trackmetrackerhealth tenant_id="<tenant_id>"
This custom command performs various operations and relies on the REST endpoint:
![screen9.png](_images/screen97.png)
![screen10.png](_images/screen107.png)
| trackme mode=post url="/services/trackme/v2/configuration/get_tenant_ops_status"
TrackMe components, such as Hybrid Trackers and their related custom commands call the components register at the backend level, and these states are stored in a Python dictionanry in the central store KVstore collection:
| inputlookup trackme_virtual_tenants | eval keyid=_key
| fields tenant_id, tenant_objects_exec_summary
![screen11.png](_images/screen1113.png)