.. _troubleshooting: .. _trackme_troubleshooting: Troubleshooting TrackMe ####################### TrackMe is heavily instrumented: every REST API handler, custom command backend and alert action writes structured, indexed logs, and a family of audit dashboards turns those logs into actionable views. When something does not behave as expected, you almost never need to guess — the answer is in the logs, and this page is the map to find it. .. _troubleshoot_logging_overview: How TrackMe logs ================ All TrackMe components log in a consistent, structured format. Events carry a ``log_level`` field (``INFO``, ``WARNING``, ``ERROR``, ``DEBUG``), and most carry contextual fields such as ``tenant_id``, ``component``, ``task``, ``run_time`` and ``instance_id`` so you can pivot quickly from a symptom to a root cause. The table below summarises where each class of events is indexed: .. list-table:: :header-rows: 1 :widths: 26 40 34 * - Log source - Sourcetype - Index * - **REST API endpoints** - ``trackme:rest_api`` - ``_internal`` * - **Custom commands** (trackers and backends) - ``trackme:custom_commands:*`` - ``_internal`` * - **Alert actions** (Notable, Auto-ack, Stateful, etc.) - ``modular_alerts:trackme_*`` - ``_internal`` and ``cim_modactions`` * - **Health events** (tracker healthy status) - ``trackme:health`` - TrackMe audit index (``trackme_audit*``) .. note:: Ingest-time parsing is carefully handled with ``TRUNCATE=0`` on the TrackMe sourcetypes, so even very large events are indexed in full and never suffer from truncation. .. _troubleshoot_logs_inspector: Logs Inspector (recommended starting point) =========================================== The **Logs Inspector** dashboard is the fastest way to inspect TrackMe activity without writing any SPL. It is available from the navigation bar under **Audit & troubleshoot > Logs Inspector**, and it unifies the REST API, custom commands and alert action logs into a single point-and-click experience. .. image:: img_v2/troubleshoot/audit-logs-img-014485@2x.png :alt: TrackMe Logs Inspector dashboard :align: center :width: 100% :class: with-border Use the filter bar at the top of the dashboard to narrow down exactly what you need: - **Time Range** — any Splunk time range, defaulting to the last 60 minutes. - **Log Source** — ``All``, ``Custom Commands`` (``trackme:custom_commands:*``), ``REST API`` (``trackme:rest_api``) or ``Modular Alerts`` (``modular_alerts:trackme_*``). The dashboard automatically selects the correct index(es) for each source. - **Logging Level** — ``All``, ``INFO``, ``ERROR``, ``WARNING`` or ``DEBUG``. - **Keyword Search** — free-text filter applied to the raw events (for example a ``tenant_id``, a component name or an error message fragment). The dashboard then renders, for the selected filters: - **Log Level Distribution** — a donut chart summarising the balance of ``INFO`` / ``WARNING`` / ``ERROR`` events, so a spike in errors is visible at a glance. - **Events by Logging Level Over Time** — a stacked column chart (5-minute spans) revealing when errors or warnings occurred. - **Logging Events** — a paginated table of the most recent events (up to 10,000), where each row can be expanded to reveal the full event, and the sourcetype, log level and command are surfaced as dedicated columns. .. tip:: Every panel provides an **Open in Search** action, which hands the underlying SPL over to the Splunk search view so you can extend the investigation with your own commands. .. _troubleshoot_reviewing_logs_search: Reviewing logs in Splunk Search =============================== When you need the full power of SPL, you can query the TrackMe logs directly. The following searches are the building blocks used throughout support investigations. .. _troubleshoot_rest_api_endpoints_logging: REST API endpoints logging --------------------------- All TrackMe REST API handlers log to a dedicated, automatically indexed log file: :: index=_internal sourcetype=trackme:rest_api Custom commands logging ----------------------- Each custom command backend (trackers and helper commands) logs to its own log file, tied to a specific sourcetype following the ``trackme:custom_commands:*`` convention. Review all of them at once with: :: index=_internal sourcetype=trackme:custom_commands:* To target a single backend, append its name, for example the Health tracker: :: index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth Alert actions logging --------------------- TrackMe ships multiple alert actions (Notable, Auto-acknowledge, Stateful alerts, Free-style REST call, and more). Each writes to its dedicated log file, and the Splunk modular alert framework additionally records execution in ``cim_modactions``. Review all TrackMe alert action logs with: :: (index=_internal OR index=cim_modactions) sourcetype=modular_alerts:trackme_* The all-in-one search --------------------- To review **everything** TrackMe logs — REST API, custom commands and alert actions — in a single search, use the combined query below. This is the same search that powers the Logs Inspector dashboard: :: (index=_internal OR index=cim_modactions) ((sourcetype=trackme:rest_api OR sourcetype=trackme:custom_commands:*) OR sourcetype=modular_alerts:trackme_*) .. image:: img_v2/troubleshoot/search-logs-img-014487@2x.png :alt: Combined TrackMe logs search in Splunk :align: center :width: 100% :class: with-border Focusing on errors ------------------ Because every TrackMe event carries a ``log_level`` field, isolating problems is straightforward. Add ``log_level=ERROR`` to any of the searches above to surface only errors, for example across all custom commands: :: index=_internal sourcetype=trackme:custom_commands:* log_level=ERROR Or, to catch errors across the whole product in one pass: :: (index=_internal OR index=cim_modactions) ((sourcetype=trackme:rest_api OR sourcetype=trackme:custom_commands:*) OR sourcetype=modular_alerts:trackme_*) log_level=ERROR .. _troubleshoot_health_events: Health events and the Health tracker ==================================== Health events ------------- TrackMe produces and indexes **health events** that track the healthy status of its trackers on a per Virtual Tenant basis. These events are stored in the TrackMe audit index under the ``trackme:health`` sourcetype and carry a ``last_status`` field (``success`` / ``failure``): :: index=trackme_audit* sourcetype=trackme:health The live, aggregated view of these statuses is surfaced by the **Audit - Operational Statuses** dashboard described below. The Health tracker ------------------ TrackMe runs an important tracker that is automatically created for each Virtual Tenant. Among other responsibilities, it is in charge of triggering **schema upgrade** procedures when TrackMe is upgraded, and of generating a number of per-tenant metrics. Its execution is logged under the ``trackmetrackerhealth`` sourcetype, with rich per-task detail (one line per task, including its run time). For example, to review the schema upgrade activity: :: index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth task="schema_upgrade" To track the run time of every task handled by the Health tracker: :: index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth instance_id=* task_instance_id=* task=* run_time=* tenant_id=* | table _time tenant_id instance_id task task_instance_id run_time _raw | sort 0 - _time .. _troubleshoot_audit_dashboards: Audit dashboards ================ Beyond the Logs Inspector, TrackMe provides a set of audit dashboards for troubleshooting and auditing specific behaviours. They are all reachable from the **Audit & troubleshoot** menu in the navigation bar. Audit - Operational Statuses ---------------------------- Shows which tenants are operational versus degraded, built from the components register and the health events. An **Overview** tab summarises the count of active and degraded tenants, the operational-status breakdown and a per-tenant status table, while a **Job component detail** tab drills into per-job component health. The status is a live snapshot; the selected time range applies to the degradation-events history: .. image:: img_v2/troubleshoot/tenant-operational-status-img-014489@2x.png :alt: Audit - Operational Statuses dashboard :align: center :width: 100% :class: with-border Audit - Trackers Performance Deep Dive -------------------------------------- Provides a comprehensive review of the trackers' run time performance — which trackers run longest and how their runtime trends over time. Filter by tenant, tracker and time range to see the most prominent trackers by average runtime: .. image:: img_v2/troubleshoot/deepdive-1-img-014490@2x.png :alt: Audit - Trackers Performance Deep Dive dashboard - prominent trackers :align: center :width: 100% :class: with-border The lower panels chart the average runtime per tracker over time and provide a per-tracker performance table with the average, p95, max and latest runtime (in seconds), making runtime regressions and outliers easy to spot: .. image:: img_v2/troubleshoot/deepdive-2-img-014491@2x.png :alt: Audit - Trackers Performance Deep Dive dashboard - runtime over time and per-tracker table :align: center :width: 100% :class: with-border Audit - KVstore Collections --------------------------- Reports the object count and on-disk size for every TrackMe KVstore collection, grouped by tenant. Use it to review the total number of collections, their aggregated size and records, and to identify the largest collections — filter by tenant or search by collection name: .. image:: img_v2/troubleshoot/audit-kvstore-collections-img-014492@2x.png :alt: Audit - KVstore Collections dashboard :align: center :width: 100% :class: with-border Audit - Data Sampling --------------------- Investigates the event-format recognition (Data Sampling) feature for the splk-dsm component (part of splk-feeds): per-object sampling status, the regex models behind it, and the executor backend. Filter by tenant, object, status and feature, then review the sampling status distribution (green / orange / red), the anomaly reasons and the detected event formats across its **Overview & status**, **Models** and **Backend & logs** tabs: .. image:: img_v2/troubleshoot/audit-data-sampling-img-014493@2x.png :alt: Audit - Data Sampling dashboard :align: center :width: 100% :class: with-border