.. _troubleshooting:

Troubleshooting
###############

REST API endpoints logging
==========================

All TrackMe REST API handlers log events to a unique log file which is automatically indexed in Splunk, available through:

::

    index=_internal sourcetype=trackme:rest_api

Ingest time parsing is carefully handled, so even large events wouldn't suffer from truncation.

You can rely on the logging level to review specific classes of events:

*review errors:*

::

    index=_internal sourcetype=trackme:rest_api log_level=ERROR

Custom commands logging
=======================

Each custom command backend available in TrackMe logs events to a dedicated log file, which itself ties to a specific sourcetype.

You can review all custom command logs from the following convention:

::

    index=_internal sourcetype=trackme:custom_commands:*

Similarly, you can review any errors such as:

::

    index=_internal sourcetype=trackme:custom_commands:* log_level=ERROR

The navigation bar provides pre-classified shortcuts per TrackMe component:

.. image:: img_v2/troubleshoot/screen1.png
   :alt: screen1.png
   :align: center
   :width: 1200px
   :class: with-border

Alert actions logging
=====================

TrackMe provides multiple alert actions, such as the Notable alert action, each alert action logs event its dedicated log file.

You can review all modular alert actions logs from the following convention:

::

    index=_internal sourcetype=modular_alerts:trackme_*

TrackMe Health events
=====================

TrackMe produces and indexes health events for the purpose of tracking its tracker healthy status, you can review these events via the sourcetype **trackme:health**:

Assuming your TrackMe audit indexe(s) all start by ``trackme_audit*``:

::

    index=trackme_audit* sourcetype=trackme:state

Health events are indexed events generated from the live statuses from the following REST endpoint:

::

    | trackme mode=post url=/services/trackme/v2/configuration/get_tenant_ops_status body="{'mode': 'raw'}" | trackmeopsstatusexpand

TrackMe Health Tracker
======================

**TrackMe has an important tracker that is automatically created on a per Virtual Tenant basis, notably this tracker is responsible for triggering upgrade procedures as needed, called schema upgrade.**

.. hint:: **TrackMe 2.1.0 improvements:**

    - Since TrackMe 2.1.0, the logging format was massivel improved regarding this very specific component, so you can track easily the execution of every single task, its run time and so forth.

*Access the logs:*

::

    index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth task="schema_upgrade"

*You can track the run time of every task handled by the Health tracker using the following search example:*

::

    index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth instance_id=* task_instance_id=* task=* run_time=* tenant_id=*
    | table _time tenant_id instance_id task task_instance_id run_time _raw
    | sort 0 - _time

Audit Dashboards
================

**Several dashboards are provided for the purposes of troubleshooting and auditing TrackMe features and behaviors:**

.. image:: img_v2/troubleshoot/screen2.png
   :alt: screen2.png
   :align: center
   :width: 1200px
   :class: with-border

Audit - Operational Statuses
----------------------------

**This dashboard provides a summary review of the Virtual Tenants operation statuses, which relies on the components register and the Health events:**

.. image:: img_v2/troubleshoot/audit_ops_status.png
   :alt: audit_ops_status.png
   :align: center
   :width: 1200px
   :class: with-border

Audit - Trackers Performance DeepDive
-------------------------------------

**This dashboard provides a comprehensive review of the Trackers run time performance, this Key Performance Indicator is generated and logged when a tracker is executed:**

.. image:: img_v2/troubleshoot/audit_perf_trackers.png
   :alt: audit_perf_trackers.png
   :align: center
   :width: 1200px
   :class: with-border

Audit - KVstore Collections
---------------------------

**This dashboard provides a summary overview of the KVstore collections classified per tenant, this allows to review the global size of the KVstore collections as well as the details per KVstore:**

.. image:: img_v2/troubleshoot/audit_kvstore.png
   :alt: audit_kvstore.png
   :align: center
   :width: 1200px
   :class: with-border

Audit - Data Sampling
---------------------

**This dashboard is investigating the status of the Data sampling feature for the splk-dsm component (part of splk-feeds):**

.. image:: img_v2/troubleshoot/audit_data_sampling.png
   :alt: audit_data_sampling.png
   :align: center
   :width: 1200px
   :class: with-border