Scheduling Virtual Tenants

Monitoring the Virtual Tenant scheduling activity

TrackMe heavily relies on the proper scheduling and execution of its related scheduled jobs, called Trackers in the context of TrackKe.

You can easily monitor the scheduliing activity from the Virtual Tenants user interface:

screen1.png screen2.png

There should ideally be no skipping searches in the TrackMe deployment, TrackMe trackers carefully log their own run time performance which are leveraged in the “TrackMe Performance Deepdive” dashboard:

screen3.png

  • Skipping searches can happen if the deployment is slightly loaded, and/or the environment is lacking of compute resources (CPUs).

  • If you have skipping searches, this generally indicates that you need to review the quality and the performances of the searches, or increases the compute resources for the TrackMe deployments.

Trackers logging and troubleshooting

All TrackMe trackers log their own activity carefully including their run time performance Key Performance Indicator, you can access the TrackMe internal logs through the navigation bar:

screen4.png

For instance, splk-feeds trackers are orchestrated by a TrackMe custom command called trackmetrackerexecutor:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerexecutor
screen5.png

In logging INFO level, the custom command logs the tenant_id as well as its related scheduling configuration, and a result summary:

screen6.png

Note: in DEBUG mode TrackMe will become very verbose, therefore the DEBUG mode should be activated only temporarily and debugging purposes

You can control the logging level in the Configuration user interface:

screen7.png

Automated scheduling re-attempting concept

TrackMe has a concept of automated re-attempts when it detects that a given execution cannot be performed due to max search concurrency being reached.

This can be observed in the custom command logs such as:

index=_internal sourcetype=trackme:custom_commands:* "temporary failure"
screen8.png

If the maxium number of re-attempts is reached, its execution will be canceled and it will log this as a permanent failure in error log level.