Operational status Virtual Tenants
##################################

What is the Ops status for Virtual Tenants
==========================================

.. hint:: **TrackMe Virtual Tenants are themselves continuously monitored to report any kind of failures that would be encountered by any of the application trackers.**

**TrackMe trackers can fail when technical issues are encountered, this can happen for various reasons, such as:**

- Overloaded environments, where there are not search slots available (although TrackMe has a concept of automated re-attempts)
- Search Head is in detention (overused file-systems, etc)
- Network connectivity issues, credentials issues
- Corrupted configuration (ex: missing Knowledge Objects or incorrect permissions)
- And many more

With the goal of providing the highest visibility and quality, every Tracker and related custom commands carefully report failures to a central component called the components register.

In addition, when a Virtual Tenant is created, a Health Tracker is created which investigates the component register results, reports and generates Health events continuously.

**You can easily observe the operation status of TrackMe Virtual Tenants in the Virtual Tenants user interface:**

.. image:: img_v2/vtenant_ops_status/screen1.png
   :alt: screen1.png
   :align: center
   :width: 1200px
   :class: with-border

**Click on the chart to open the detailed view:**

.. image:: img_v2/vtenant_ops_status/screen2.png
   :alt: screen2.png
   :align: center
   :width: 1200px
   :class: with-border

**Click on "Open in search" to access detailed per tracker results in the Splunk Search user interface:**

.. image:: img_v2/vtenant_ops_status/screen3.png
   :alt: screen3.png
   :align: center
   :width: 1200px
   :class: with-border

**Health events are stored in the audit indexe(s) of the Virtual Tenant(s), you can search for Health failures events easily, such as:**

::

    index=trackme_audit sourcetype="trackme:health" last_status="failure"
    | timechart count by tenant_id

.. image:: img_v2/vtenant_ops_status/screen4.png
   :alt: screen4.png
   :align: center
   :width: 1200px
   :class: with-border

Example of failure detection
============================

In this example, a remote Splunk Deployment is encountering issues due to a file-system being filled over the accepted limit by Splunk, therefore this instance does not accept searches any longer.

**We can observe the Vetants Ops status reporting issues currently, as well as which Virtual Tenants are affected currently:**

.. image:: img_v2/vtenant_ops_status/screen5.png
   :alt: screen5.png
   :align: center
   :width: 1200px
   :class: with-border

**Accessing the detailed Ops status view quickly shows the affected tenants, related accounts and technical root cause:**

.. image:: img_v2/vtenant_ops_status/screen6.png
   :alt: screen6.png
   :align: center
   :width: 1200px
   :class: with-border

**We can access the live status from REST:**

.. image:: img_v2/vtenant_ops_status/screen7.png
   :alt: screen7.png
   :align: center
   :width: 1200px
   :class: with-border

Clearing the Virtual Tenants Operational Status
===============================================

**From TrackMe 2.0.97, you can clear the Virtual Tenants Operational Status via the Virtual Tenants UI and from the REST API:**

*Access to the screen TrackMe Tenants Operational health statuses:*

.. image:: img_v2/vtenant_ops_status/screen_clear_status.png
   :alt: screen_clear_status.png
   :align: center
   :width: 1200px
   :class: with-border

*You can then request to clear for all tenants, or a selection of tenants and you can also choose to execute or not the health tracker during the clear process:*

.. image:: img_v2/vtenant_ops_status/screen_clear_status2.png
   :alt: screen_clear_status2.png
   :align: center
   :width: 1200px
   :class: with-border

*Underneath, the UI calls the REST endpoint to clear the Virtual Tenants Operational Status:*

.. image:: img_v2/vtenant_ops_status/screen_clear_status3.png
   :alt: screen_clear_status3.png
   :align: center
   :width: 1200px
   :class: with-border

*Example:*

::

   | trackme url="/services/trackme/v2/vtenants/admin/clear_exec_summary" mode="post" body="{'tenants': 'all'}"

Additional resources
====================

An additional (UFS) dashboard is available at:

- Navigation Bar / Audit & Troubleshoot / Audit - Operational Statuses

.. image:: img_v2/vtenant_ops_status/screen8.png
   :alt: screen8.png
   :align: center
   :width: 1200px
   :class: with-border

Virtual Tenants Ops status technical overview
=============================================

**When a Virtual Tenant is created, a Health tracker is created and executed every 5 minutes:**

::

    trackme_health_tracker_tenant_<tenant_id>

**This tracker executes the following custom command:**

::

    | trackmetrackerhealth tenant_id="<tenant_id>"

**This custom command performs various operations and relies on the REST endpoint:**

.. image:: img_v2/vtenant_ops_status/screen9.png
   :alt: screen9.png
   :align: center
   :width: 1200px
   :class: with-border

.. image:: img_v2/vtenant_ops_status/screen10.png
   :alt: screen10.png
   :align: center
   :width: 1200px
   :class: with-border

::

    | trackme mode=post url="/services/trackme/v2/configuration/get_tenant_ops_status"

**TrackMe components, such as Hybrid Trackers and their related custom commands call the components register at the backend level, and these states are stored in a Python dictionanry in the central store KVstore collection:**

::

    | inputlookup trackme_virtual_tenants | eval keyid=_key
    | fields tenant_id, tenant_objects_exec_summary

.. image:: img_v2/vtenant_ops_status/screen11.png
   :alt: screen11.png
   :align: center
   :width: 1200px
   :class: with-border