Troubleshooting TrackMe

TrackMe is heavily instrumented: every REST API handler, custom command backend and alert action writes structured, indexed logs, and a family of audit dashboards turns those logs into actionable views. When something does not behave as expected, you almost never need to guess — the answer is in the logs, and this page is the map to find it.

How TrackMe logs

All TrackMe components log in a consistent, structured format. Events carry a log_level field (INFO, WARNING, ERROR, DEBUG), and most carry contextual fields such as tenant_id, component, task, run_time and instance_id so you can pivot quickly from a symptom to a root cause.

The table below summarises where each class of events is indexed:

Log source	Sourcetype	Index
REST API endpoints	trackme:rest_api	_internal
Custom commands (trackers and backends)	trackme:custom_commands:*	_internal
Alert actions (Notable, Auto-ack, Stateful, etc.)	modular_alerts:trackme_*	_internal and cim_modactions
Health events (tracker healthy status)	trackme:health	TrackMe audit index (trackme_audit*)

Note

Ingest-time parsing is carefully handled with TRUNCATE=0 on the TrackMe sourcetypes, so even very large events are indexed in full and never suffer from truncation.

Logs Inspector (recommended starting point)

The Logs Inspector dashboard is the fastest way to inspect TrackMe activity without writing any SPL. It is available from the navigation bar under Audit & troubleshoot > Logs Inspector, and it unifies the REST API, custom commands and alert action logs into a single point-and-click experience.

Use the filter bar at the top of the dashboard to narrow down exactly what you need:

• Time Range — any Splunk time range, defaulting to the last 60 minutes.

• Log Source — All, Custom Commands (trackme:custom_commands:*), REST API (trackme:rest_api) or Modular Alerts (modular_alerts:trackme_*). The dashboard automatically selects the correct index(es) for each source.

• Logging Level — All, INFO, ERROR, WARNING or DEBUG.

• Keyword Search — free-text filter applied to the raw events (for example a tenant_id, a component name or an error message fragment).

The dashboard then renders, for the selected filters:

• Log Level Distribution — a donut chart summarising the balance of INFO / WARNING / ERROR events, so a spike in errors is visible at a glance.

• Events by Logging Level Over Time — a stacked column chart (5-minute spans) revealing when errors or warnings occurred.

• Logging Events — a paginated table of the most recent events (up to 10,000), where each row can be expanded to reveal the full event, and the sourcetype, log level and command are surfaced as dedicated columns.

Tip

Every panel provides an Open in Search action, which hands the underlying SPL over to the Splunk search view so you can extend the investigation with your own commands.

Reviewing logs in Splunk Search

When you need the full power of SPL, you can query the TrackMe logs directly. The following searches are the building blocks used throughout support investigations.

REST API endpoints logging

All TrackMe REST API handlers log to a dedicated, automatically indexed log file:

index=_internal sourcetype=trackme:rest_api

Custom commands logging

Each custom command backend (trackers and helper commands) logs to its own log file, tied to a specific sourcetype following the trackme:custom_commands:* convention. Review all of them at once with:

index=_internal sourcetype=trackme:custom_commands:*

To target a single backend, append its name, for example the Health tracker:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth

Alert actions logging

TrackMe ships multiple alert actions (Notable, Auto-acknowledge, Stateful alerts, Free-style REST call, and more). Each writes to its dedicated log file, and the Splunk modular alert framework additionally records execution in cim_modactions. Review all TrackMe alert action logs with:

(index=_internal OR index=cim_modactions) sourcetype=modular_alerts:trackme_*

The all-in-one search

To review everything TrackMe logs — REST API, custom commands and alert actions — in a single search, use the combined query below. This is the same search that powers the Logs Inspector dashboard:

(index=_internal OR index=cim_modactions) ((sourcetype=trackme:rest_api OR sourcetype=trackme:custom_commands:*) OR sourcetype=modular_alerts:trackme_*)

Focusing on errors

Because every TrackMe event carries a log_level field, isolating problems is straightforward. Add log_level=ERROR to any of the searches above to surface only errors, for example across all custom commands:

index=_internal sourcetype=trackme:custom_commands:* log_level=ERROR

Or, to catch errors across the whole product in one pass:

(index=_internal OR index=cim_modactions) ((sourcetype=trackme:rest_api OR sourcetype=trackme:custom_commands:*) OR sourcetype=modular_alerts:trackme_*) log_level=ERROR

Health events and the Health tracker

Health events

TrackMe produces and indexes health events that track the healthy status of its trackers on a per Virtual Tenant basis. These events are stored in the TrackMe audit index under the trackme:health sourcetype and carry a last_status field (success / failure):

index=trackme_audit* sourcetype=trackme:health

The live, aggregated view of these statuses is surfaced by the Audit - Operational Statuses dashboard described below.

The Health tracker

TrackMe runs an important tracker that is automatically created for each Virtual Tenant. Among other responsibilities, it is in charge of triggering schema upgrade procedures when TrackMe is upgraded, and of generating a number of per-tenant metrics.

Its execution is logged under the trackmetrackerhealth sourcetype, with rich per-task detail (one line per task, including its run time). For example, to review the schema upgrade activity:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth task="schema_upgrade"

To track the run time of every task handled by the Health tracker:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth instance_id=* task_instance_id=* task=* run_time=* tenant_id=*
| table _time tenant_id instance_id task task_instance_id run_time _raw
| sort 0 - _time

Audit dashboards

Beyond the Logs Inspector, TrackMe provides a set of audit dashboards for troubleshooting and auditing specific behaviours. They are all reachable from the Audit & troubleshoot menu in the navigation bar.

Audit - Operational Statuses

Shows which tenants are operational versus degraded, built from the components register and the health events. An Overview tab summarises the count of active and degraded tenants, the operational-status breakdown and a per-tenant status table, while a Job component detail tab drills into per-job component health. The status is a live snapshot; the selected time range applies to the degradation-events history:

Audit - Trackers Performance Deep Dive

Provides a comprehensive review of the trackers’ run time performance — which trackers run longest and how their runtime trends over time. Filter by tenant, tracker and time range to see the most prominent trackers by average runtime:

The lower panels chart the average runtime per tracker over time and provide a per-tracker performance table with the average, p95, max and latest runtime (in seconds), making runtime regressions and outliers easy to spot:

Audit - KVstore Collections

Reports the object count and on-disk size for every TrackMe KVstore collection, grouped by tenant. Use it to review the total number of collections, their aggregated size and records, and to identify the largest collections — filter by tenant or search by collection name:

Audit - Data Sampling

Investigates the event-format recognition (Data Sampling) feature for the splk-dsm component (part of splk-feeds): per-object sampling status, the regex models behind it, and the executor backend. Filter by tenant, object, status and feature, then review the sampling status distribution (green / orange / red), the anomaly reasons and the detected event formats across its Overview & status, Models and Backend & logs tabs: