Rollback

Danger

A rollback is a restore, not an un-install.

Schema migrations are forward-only. Reinstalling an older package reverts the code and none of the migrated data, over collections and knowledge objects that the migrations already reshaped, and the schema marker is silently realigned so that the Tenants Update statuses screen reports updated. The reliable rollback is to restore the pre-upgrade backups on the reinstalled package.

Before deciding, separate the two kinds of change listed in the behaviour changes table of Validating after an upgrade. Alerting behaviour (impact-score weights, the score contributed by an outlier, policy cadence, threshold locks, per-entity maintenance) is configurable: if the driver is alert-volume surprises or a scoring or policy outcome, tune on the current release first, a rollback is not needed. Changes that are not configurable (the native ML engine, the schema shape, the rebuilt user interface) are the only ones a rollback actually reverts; weigh them against its cost. A rollback across the native ML engine migration (2.3.15) additionally requires a full retraining of every Outliers model, so the models are inactive until it completes.

Procedure

  1. Decide against fresh evidence. Capture the post-upgrade evidence first (update-status screenshot, model inventory, Ops Status) so that the comparison stays possible, and confirm the driver is not addressable by configuration on the current release.

  2. Freeze changes and take a final backup. A TrackMe backup at the current release, plus, on Splunk Enterprise, copies of the application directory and of the KVstore, before any package operation.

  3. Reinstall the previous release in place. Same procedure as step 2.1 of the Upgrade procedure with the previous package. Never remove the application directory: local/ holds every tenant, tracker and credential.

  4. Restore the pre-upgrade state. Restore the pre-upgrade KVstore backup (splunk restore kvstore) and the pre-upgrade local/ from the application-directory copy taken in step 1.2 of the procedure. This returns every collection, transform and tracker to their pre-upgrade shape. On Splunk Cloud, or when those copies are not available, restore the pre-upgrade TrackMe backup from Backup & Restore (see Backup & restore) as a whole run, so that every tenant archive and the global archive are restored together (restoring a single archive leaves tenant and global state inconsistent; releases before the multi-archive format produce one archive per run, which is restored as such). Run the restore dry run first and review its preview before the real restore. Then apply the manual compensation below.

  5. Fallback only: compensate the native ML engine migration by hand. If the rollback crosses release 2.3.15 and the KVstore was not restored, rewrite every model’s algorithm back to the MLTK engine (see below) and retrain all models. Until retraining completes, Outliers are inactive.

  6. Re-validate. Steps 1.1, 3.1 and 3.2 of the procedure, with the old behaviour expected (boolean alerting and no score events when rolling back below 2.3.5). The update-status screen will read updated; it cannot see a downgrade. Validate the data.

  7. Report to TrackMe Support. A rollback must be understood before the next attempt; send the validation record and the migration logs (see Support).

The native ML engine compensation (fallback only)

Migration 2315 rewrote every Outliers model to algorithm=TrackMeNativeDensityFunction and set model_storage=kvstore. Code earlier than 2.3.15 has no such engine and does not validate the value: the training search fails at search time and models stay silently inactive until reverted. Detect, per tenant and component:

| inputlookup trackme_<component>_outliers_entity_rules_tenant_<tenant_id>
| rex max_match=0 field=entities_outliers "\"algorithm\": \"(?<algorithm>[^\"]+)\""
| stats count by algorithm

Rewrite both fields the migration changed, algorithm back to DensityFunction and model_storage back to file (back up the collection first; the key-preserving update keeps the records in place):

| inputlookup trackme_<component>_outliers_entity_rules_tenant_<tenant_id> | eval keyid=_key
| rex field=entities_outliers mode=sed "s/\"algorithm\": \"TrackMeNativeDensityFunction\"/\"algorithm\": \"DensityFunction\"/g"
| rex field=entities_outliers mode=sed "s/\"model_storage\": \"kvstore\"/\"model_storage\": \"file\"/g"
| outputlookup trackme_<component>_outliers_entity_rules_tenant_<tenant_id> append=t key_field=keyid

The stored model searches that the migration reset to pending are regenerated by the training run that follows; the fitted native models in kv_trackme_native_ml_models_tenant_<tenant_id> are simply ignored by the older release.

Then run ML Train across the affected models (Tenant Home → Bulk edit → Outliers Detection Actions, or the mltrain reports) and confirm the Outliers charts render again.

What a rollback does not undo without the KVstore restore

Migrated state

Behaviour on the older release

New KVstore collections (native models, score cache, shadow cache, intent locks, labels, maintenance and others)

Ignored: the older code never reads them. Dormant, not harmful. They re-activate on a later re-upgrade.

New fields on existing records; refreshed transforms

Carried along untouched; the older code reads the fields it knows.

Rewritten TrackMe-managed alert SPL and tracker definitions touched by migrations

Remain in their migrated form in local/. This is the reason the pre-upgrade local/ copy matters.

Model algorithm and model_storage values (migration 2315)

Actively incompatible below 2.3.15: the one compensation that must be performed by hand if the KVstore is not restored.

Schema marker

Silently realigned to the running version; the update-status screen reports updated (see Schema migrations).

Hint

Why retraining is part of every rollback across 2.3.15

The two releases fit models with different engines in different stores, so a retrain on the rolling window is unavoidable, and freshly retrained bounds legitimately differ because the window has moved. The pre-upgrade evidence (step 1.3 of the procedure) makes this explainable instead of surprising.