Backup & restore

TrackMe backs up its KV-store state and knowledge objects to compressed archives, and restores them per tenant, per whole run, or selectively. This page is an overview — the Backup & Restore white paper has the full step-by-step procedure.

How backups are structured

A backup run produces one archive per tenant plus one global archive, rather than a single monolithic file (which became unrestorable at multi-gigabyte scale). Each run has a RUN_ID and a manifest listing every archive and its checksum:

  • a per-tenant archive holds that tenant’s collections, knowledge objects, and account record;

  • the global archive holds everything else — the tenant registry, maintenance, bank holidays, license, ML models, and Guardian state.

Crucially, the archives are independent: a single corrupt archive is marked failed and does not abort the rest of the run, a restore, or a delete.

Restoring

There are three restore modes — a single archive, a whole run, or a legacy flat archive from before the multi-archive format — and you can selectively restore just specific collections or knowledge objects.

Important

Always run a restore as a dry run first, review what it will change, then run it for real. A single-tenant restore overwrites that tenant’s live state by design — coordinate with operators, ideally inside a maintenance window.

Scheduling, storage, and integrity

  • A scheduled backup runs daily and is enabled automatically once you create your first tenant, with a retention sweep.

  • Every archive is integrity-checked when produced, and you can validate one before a critical restore.

  • The backup directory and retention (in days) are configurable under Configuration → System settings → Backup & Restore (see below), with a fail-safe fallback to the default location (see Backup storage — retention and location (2.4.3)).

  • Two Configuration Guardian checks watch this — one for an archive that has grown too old, and one for a run that is missing tenants.

Where the backup settings live

The location, retention and S3 export target are all set from Configuration → System settings → Backup & Restore. The section holds three settings — Backup retention (days), Backup directory (optional) and S3 Export account (enables S3 upload) — and, like every System setting, each carries an info (ⓘ) button explaining what it does, its default, and when to change it:

config-system-backups-welcome-img-014612@2x.png

Backup location. By default, archives are written to $SPLUNK_HOME/etc/apps/trackme/backup. Set Backup directory (optional) to an absolute path (env vars and ~ are expanded) to write them elsewhere — a larger volume, an NFS mount, or a path covered by your infrastructure backup tooling. If the directory cannot be created or written to, TrackMe logs an error, raises the backup_directory_unwritable Configuration Guardian alert, and falls back to the default so backups never silently stop. On a Search Head Cluster the path must be writable on every member, and changing it affects new backups only.

config-system-backups-location-img-014613@2x.png

Backup retention. Backup retention (days) (default 30) controls the age-based purge of local archives, covering both modern per-tenant/global runs and legacy flat archives. Raise it if your disaster-recovery policy requires keeping restore points further back; lower it to save disk space (for example when archives are also exported to S3).

config-system-backups-retention-img-014615@2x.png

Off-site export to S3

From 2.4.3, completed archives can be exported off-site to S3 — Amazon S3 or any S3-compatible store (MinIO, Ceph, …) — for a disaster-recovery copy outside the Splunk deployment, with no external scripting. It is:

  • Opt-in — nothing is uploaded until you create an S3 Export account (under Configuration → Manage S3 export accounts) and select it in the Backup & Restore settings. Connectivity is verified on save, so a broken target cannot be saved silently.

  • Fail-open — if S3 is unreachable the backup still completes and the local archives are always intact; the upload failure is recorded per archive, never raised.

  • Additive — S3 export is an extra copy of the same archives, uploaded after each run.

Once you have created an S3 Export account, enable off-site upload by selecting it in the S3 Export account (enables S3 upload) dropdown of Configuration → System settings → Backup & Restore. Leaving it on (none) keeps S3 export disabled; selecting an account uploads every completed backup archive to that target from the next run onwards:

config-system-backups-export-s3-img-014617@2x.png

TrackMe does not delete objects from the bucket — S3-side retention is left to a bucket lifecycle policy. A third Configuration Guardian check, s3_export_failing, surfaces an unreachable target or a run that failed to upload. The full setup is in the white paper.

See also