Upgrading TrackMe

1. Introduction

Upgrading TrackMe is a straightforward process which is similar to the installation.

However, TrackMe as well implements a sophisticated and automated upgrade process called schema version which is used to dynamically run specific action depending on the origin versoon and the state of TrackMe Virtual tenants.

The action can imply the deletion, creation or update of TrackMe knowledge objects depending on the context.

The purpose of this documentation is to deeply describe how the schema version upgrade works.

2. Schema version

Once you have upgraded TrackMe using the Splunk standard upgrade process depending on your context (Splunk Enterprise versus Cloud, SHC versus standalone), TrackMe will automatically verify and perform additional upgrade procedures as needed.

The schema version is versioning information which is stored in the KVstore record of the Virtual Tenant, you can verify the current schema version value in the central KVstore:

| inputlookup trackme_virtual_tenants | eval keyid=_key
| fields keyid, tenant_id, schema_version
screen1.png

The schema version corresponds to the TrackMe version minus the “dots”, if you are running version 2.1.0, the schema version is 2100. (yes we added a zero here, so we follow a 4 digits convention)

Hint

TrackMe 2.1.0 automated backup improvements:

  • Since TrackMe 2.1.0, the schema upgrade process was slightly improved and notably a backup process procedure was added to ensure that the upgrade process is more secure and reliable.

  • In short, if TrackMe detectes that the schema version needs to be upgraded, it will automatically execute a backup job before starting the upgrade process.

2.1 Schema version verification

The schema version verification is an automated process handled by the Health tracker of the Virtual Tenant.

When a Virtual Tenant is created, TrackMe also creates a dedicated Health tracker for this tenant, the Health tracker is responsible for various administration tasks such as detecting any issues encountered by the tenant’s trackers, as well as running the schema version verification.

Health tracker name in Splunk:

  • trackme_health_tracker_tenant_<tenant_id>

2.2 Logs

The logs of the Health tracker and any upgrade related events are available via the navigation bar shortcut “Audit & Troubleshoot / Logs - TrackMe custom commands / common to components / trackmetrackerhealth” and match the following index/sourcetype:

Log structure changes since TrackMe 2.1.0:

  • Since TrackMe 2.1.0, changes were made to improve the usability of the logs, notably a concept of task_instance_id and task_name were introduced to slightly improve tracking the activity of the tasks in the Heath tracker.

  • The field context has been replaced by a consist new field task

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth task="schema_upgrade"

You can track the run time of every task handled by the Health tracker using the following search example:

index=_internal sourcetype=trackme:custom_commands:trackmetrackerhealth instance_id=* task_instance_id=* task=* run_time=* tenant_id=*
| table _time tenant_id instance_id task task_instance_id run_time _raw
| sort 0 - _time

2.3 Schema version upgrade & traces

When the Health Tracker runs (every 5 minutes), it verifies the schema version:

2024-09-11 16:46:18,938 INFO trackmetrackerhealth.py generate 488 tenant_id="feeds-tracking", instance_id=879a5021-3abc-47f0-a6f9-e21714240537, task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, starting task.
2024-09-11 16:46:18,953 INFO trackme_libs_schema.py trackme_schema_get_version 127 task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, tenant_id="feeds-tracking", trackme_schema_get_version, current schema_version="2097"

If an upgrade is required, the following message is logged:

2024-09-11 16:46:18,953 INFO trackmetrackerhealth.py generate 592 tenant_id="feeds-tracking", instance_id=879a5021-3abc-47f0-a6f9-e21714240537, task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, detected migration required for schema version 2098, schema_version="2097", schema_version_required="2100", processing now.

Required procedures are then performed automatically, TrackMe starts by executing a backup job:

2024-09-11 16:46:31,698 INFO trackmetrackerhealth.py generate 613 tenant_id="feeds-tracking", instance_id=879a5021-3abc-47f0-a6f9-e21714240537, task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, backup post call executed successfully

It will then execute the required upgrade procedures:

2024-09-11 16:46:31,755 INFO trackme_libs_schema.py trackme_schema_upgrade_2098 8596 task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, Starting function trackme_schema_upgrade_2098, tenant_id="feeds-tracking"
2024-09-11 16:46:32,902 INFO trackme_libs_schema.py trackme_schema_upgrade_2098 8724 task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, schema migration 2098, tenant_id="feeds-tracking", successfully deleted transform definition, transform="trackme_dsm_tenant_feeds-tracking"

Once, all required upgrade operations have been achieved, the following message is logged:

2024-09-11 16:46:45,134 INFO trackme_libs_schema.py trackme_schema_update_version 225 task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, tenant_id="feeds-tracking", schema version upgraded successfully, new schema_version="2100"
2024-09-11 16:46:45,135 INFO trackmetrackerhealth.py generate 664 tenant_id="feeds-tracking", instance_id=879a5021-3abc-47f0-a6f9-e21714240537, task="schema_upgrade", task_instance_id=80a9366d-177e-465a-b065-573dfaf400b0, run_time="26.2", task has terminated.

Disabled Virtual Tenants

  • Disabled Virtual Tenants are not migrated as long as these are disabled (since the Health Tracker will not run)

  • If the Virtual Tenant is re-enabled, its upgrade will be processed automatically within the next 5 minutes