Backing up and Restoring TrackMe

Hint

TrackMe 2.3.23: multi-archive backup format and a streamlined restore UI

• From release 2.3.23, every backup run produces N+1 independent archives: one global archive plus one tenant archive per enabled Virtual Tenant.

• Each archive can be restored, downloaded, exported or deleted on its own — a failure on one archive never blocks the others.

• Restoring is driven entirely from the TrackMe Backup and Restore UI (menu API & Tooling → Backup & Restore), including a built-in dry-run preview and an explicit “this will overwrite live data” confirmation step.

• Search Head Cluster (SHC) support is transparent — TrackMe automatically delegates each operation to the SH peer that owns the archive.

• Legacy archives produced by 2.3.21 and earlier remain fully restorable (see Legacy archive compatibility (pre-2.3.22)).

Hint

TrackMe 2.4.3: configurable backup storage and off-site S3 export

• The backup directory and retention are now configurable under Configuration → System settings → Backup & Restore, with a fail-safe fallback to the default location (see Backup storage — retention and location (2.4.3)).

• Completed archives can be exported off-site to S3 — Amazon S3 or any S3-compatible store (MinIO, Ceph, …) — opt-in and fail-open (an upload failure never blocks or fails the backup; the local archives are always intact). See Off-site export to S3 (2.4.3).

Note

What TrackMe backs up:

• Knowledge Objects created by TrackMe (KVstore definitions, transforms, macros, reports, alerts, saved searches).

• KVstore collection records — TrackMe stores its state, entities and configuration in KVstore collections.

• Virtual Tenant accounts — the per-tenant vtenant_account records that drive RBAC, scheduling, component enablement, etc.

Note

Limitations:

• Disabled Virtual Tenants are not backed up. When a backup is restored, any tenant that was disabled at backup time is not restored either. If the tenant exists at restore time but was not in the backup, TrackMe force-purges it.

• Alert actions: TrackMe restores alerts that were created via TrackMe, but only TrackMe built-in alert actions and Splunk email alert actions are supported. External alert actions (custom modular alerts) must be re-configured manually after restore.

What’s new in 2.3.23 — the multi-archive model

Earlier releases produced a single monolithic .tgz archive containing every tenant and every system collection. At scale, that single file became impractical to restore reliably — a corruption in one tenant could block recovery of all the others.

From 2.3.23, each backup run is decomposed into independent archives:

• One global archive — trackme-backup-<RUN_ID>-global.tar.zst — contains the non-tenant KV collections (kv_trackme_virtual_tenants, maintenance mode, bank holidays, native ML models, license, etc.) and global knowledge objects.

• One tenant archive per enabled tenant — trackme-backup-<RUN_ID>-tenant-<tid>.tar.zst — contains that tenant’s _tenant_<tid> KV collections, its tenant-scoped knowledge objects and the tenant’s vtenant_account snapshot.

The RUN_ID has the form YYYYMMDD-HHMMSS-<6 chars> and groups every archive produced by one invocation.

Why this matters in practice:

• Per-tenant recovery. When a single tenant’s data is corrupted, you can restore just that tenant’s archive — every other tenant stays untouched.

• Per-archive isolation. A failure on one archive (zstd corruption, disk issue, etc.) does not abort the run. The other archives are produced, validated and restorable independently.

• Per-archive validation. Each archive is sha256-checksummed and zstd-tested at creation time, so corruption is caught on the producing peer rather than at restore.

• No opt-in. Once you upgrade to 2.3.23, scheduled backups produce the new format automatically.

Accessing the TrackMe Backup and Restore user interface

The Backup and Restore page is reached from the API & Tooling menu. The top section shows the global state — how many archives exist, when the most recent one was produced, and quick actions:

From here you can:

• Create backup — trigger a backup run on demand. Produces one global + one per-tenant archive on the local SH (or on whichever SH the request lands on).

• Download a backup archive — pick a run, then pick one archive within the run, then download. SHC-transparent: the file is fetched from the owning peer automatically.

• Import a backup archive — upload a .tar.zst (3.0.0) or legacy .tgz (pre-2.3.22) file. TrackMe validates it and registers it as an available archive on the receiving peer.

• Backup archive info collection — flat table of every known archive with dry-run inspect, download, delete and “Show Details” actions.

Inspecting an archive before restoring

Two options are available depending on how much detail you need:

Quick check — Dry-run inspect from the table:

The Dry-run inspect button on each row of the backup archive info table runs the archive through extraction and metadata parsing without applying any change. Use it to confirm the archive opens cleanly before triggering a real restore.

Detailed breakdown — Show Details on an archive:

The Show Details action expands an archive and lists every KVstore collection it contains with its on-disk size. Helpful for sanity-checking what is actually inside the archive, and for spotting unexpectedly small or large collections before a restore.

For SPL-driven inspection (CI checks, scripted audits), the built-in command still works against both legacy and 3.0.0 archives:

| trackmecheckbackups archives_list="*"

Restoring TrackMe — UI walkthrough

The recommended workflow lives entirely in the UI. The example below walks through restoring a single tenant from a multi-archive run — the typical use case when one tenant has been corrupted or accidentally cleared.

Step 1 — Pick a backup run

The middle section of the page lists every known backup run:

Each row shows the run ID, when it started, when it finished, how many archives it contains (e.g. 20 tenants + 1 global) and which SHC peer(s) produced it. Click Show details & run a restore on the run you want to use.

Step 2 — Pick a scope (single archive vs. whole run)

The run expands into its per-archive table:

You now have two choices:

• Restore tenant only (per row) — restores just that tenant’s archive. The recommended path when only one tenant is affected.

• Run a full restore (top right) — restores every archive in the run, both the global archive and every tenant archive. Reserve for full disaster recovery.

Step 3 — Configure the restore

Clicking Restore tenant only opens the restore modal pre-populated with the target archive’s filename:

The toggles let you decide what to restore:

• Restore KV collections — write back the KVstore records contained in the archive.

• Clean empty collections during restore — when a collection was empty at backup time, empty the live collection too. Leave on unless you have a specific reason to keep stale data.

• Restore non-tenants collections (global archive) — only applies when restoring the global archive; controls whether system-wide collections (kv_trackme_virtual_tenants, maintenance mode, bank holidays, etc.) are touched.

• Restore knowledge objects — write back the saved searches, macros, transforms and reports captured in the archive.

• Replace existing knowledge objects — when on, an existing KO with the same name is deleted and re-created from the archive. When off, the existing live KO wins.

• Restore vtenant_account record — write back the vtenant_account JSON for the tenant (component enablement, scheduling, RBAC roles, …).

• Restore tenant main KV record (missing-tenant guard) — when on, if the tenant is absent from kv_trackme_virtual_tenants at restore time, TrackMe auto-recreates the tenant record from the archive’s own snapshot. Leave on — this is what makes per-tenant restore of a deleted tenant possible.

The blocklists at the bottom let you exclude specific KV collections or specific knowledge objects by name (comma-separated). Useful for fine-grained surgical restores (“restore everything except these two collections”).

Step 4 — Restrict which collections / knowledge objects to restore (optional)

Scrolling down reveals the Customize selection panel:

By default every collection and every knowledge object in the archive is selected. To exclude items, open the multi-select dropdowns and uncheck what you want to skip:

A search box on each dropdown helps when an archive contains hundreds of collections. Clear all lets you start from an empty selection and pick just the few you want.

Step 5 — Run a dry-run preview

Always click Run dry-run preview before applying anything. The preview output shows exactly what the restore would do:

{
  "response": "TrackMe 3.0.0 restore completed: ok=1, partial=0, failed=0",
  "dry_run": true,
  "backup_run_id": "...",
  "archives": [ ... per-archive summary ... ]
}

Read the per-archive summary carefully: how many records would be loaded into each KVstore collection, which knowledge objects would be replaced, whether the missing-tenant safety guard would fire. No live data is touched at this stage — the dry-run is safe to run repeatedly while you tune the selection.

Step 6 — Apply the restore

Once the dry-run output looks right, tick the “I understand this will overwrite live data” checkbox. Only then does the Apply restore button activate:

The button is rendered in red on purpose — applying a restore is a destructive operation on the selected scope.

Step 7 — Monitor progress

The restore runs asynchronously. The modal switches to a live progress view:

The panel shows the job ID, the current status and the step in progress. The restore is resumable: if the underlying splunkd subprocess is recycled during a long run, TrackMe’s watchdog auto-resumes from the last checkpoint — you do not need to re-trigger the restore manually.

Cancel job is available if you need to abort. Cancel takes effect at the next checkpoint between archive steps, not in the middle of a single step.

Step 8 — Read the result

When the restore terminates, the modal shows the result panel:

What to look at:

• status=completed in the Restore finished panel.

• The green ok=N, partial=N, failed=N summary in the Restore result panel. ok is the number of archives that fully succeeded.

• Per-archive boxes for any archive that ended in partial or failed state, with the per-archive error included inline.

• The “Tenant record auto-recreated from archive (missing-tenant safety guard)” annotation is normal when the target tenant was missing from kv_trackme_virtual_tenants at restore time — TrackMe rebuilt the central record from the archive’s own snapshot.

• Full response JSON (for diagnostics) — expandable, contains the complete per-step structured response. Useful when raising a support ticket.

Restoring a full backup run

The Run a full restore button (visible after expanding a run, top-right of the per-archive table) opens the same modal but pre-targets the whole run: every tenant archive plus the global archive. The same toggles, customize-selection and dry-run preview apply.

Use this path when:

• You are restoring into a fresh deployment.

• You are running a disaster-recovery drill.

• Multiple tenants have been corrupted or the system-wide collections are damaged.

For single-tenant corruption, prefer Restore tenant only — the blast radius is much smaller.

Automated backups

TrackMe runs backups automatically in two situations:

• Scheduled backup — the saved search trackme_backup_scheduler runs every night and produces a fresh backup run (one global + one per enabled tenant archive).

• Schema-upgrade backup — when a TrackMe release upgrade is applied and the first Virtual Tenant goes through the schema upgrade, TrackMe produces a backup run beforehand. The run comment records the version transition.

Backup storage — retention and location (2.4.3)

Retention and the on-disk location of backup archives are configured under Configuration → System settings → Backup & Restore — the retention (in days), the backup directory and the active S3 export account.

Backup retention

Backup retention (days) ([backup_restore] → backup_retention_days, default 30) controls the age-based purge of local backup archives. The retention purge removes both 3.0.0 multi-archive runs and legacy archives older than the threshold.

On-disk location

By default, TrackMe stores backup archives under $SPLUNK_HOME/etc/apps/trackme/backup/. From 2.4.3 this is configurable with Backup directory (optional) ([backup_restore] → backup_directory):

• Leave it empty to use the default. To override, provide an absolute path (e.g. /data/backups) — relative paths are rejected. Environment variables and ~ are expanded.

• Validated at use. TrackMe creates the directory if needed and write-probes it (write + flush + fsync). If the configured directory cannot be created or written to, TrackMe logs an error, raises a backup_directory_unwritable Configuration Guardian alert, and falls back to the default location — so backups never silently stop.

• New backups only. Changing the directory affects new runs; existing archives are not moved or re-keyed.

Note

Search Head Cluster: point backup_directory at a path that exists and is writable by splunkd on every SHC peer (a shared mount, or an identical local path on each peer). Each peer writes the archives it produces to this location.

Off-site export to S3 (2.4.3)

From release 2.4.3, completed backup archives can be uploaded off-site to S3 — either Amazon S3 or any S3-compatible object store (MinIO, Ceph, …). This gives you a disaster-recovery copy outside the Splunk deployment without any external scripting.

Note

Why this is safe to enable

• Opt-in — nothing is uploaded until you configure an S3 Export account and select it in the Backup & Restore settings.

• Fail-open — if S3 is unreachable, the backup still completes; the local archives are always intact. The upload failure is recorded per-archive and surfaced (see Monitoring failures below), never raised.

• Local backups are unchanged — S3 export is an additional copy of the same archives, uploaded after each run completes.

Step 1 — Create an S3 Export account

S3 targets are managed from a dedicated page reachable from the top navigation: Configuration → Manage S3 export accounts. The page lists every target with its bucket, endpoint and last connectivity status, and offers per-row Edit / Clone / Test connectivity / Delete actions plus Test all and Add account. When no target exists yet, the page shows a short getting-started guide:

Click Add account and name a bucket, region, prefix and (for S3-compatible stores) an endpoint:

Key fields:

• Bucket name — the destination bucket (it must already exist).

• Region — the AWS region. Optional for S3-compatible stores; defaults to us-east-1 for request signing.

• Prefix — an optional key prefix. Archives are stored under <prefix>/<run_id>/<filename>.

• Endpoint URL — leave empty for AWS S3; set it (e.g. https://s3.example.com) to target an S3-compatible store such as MinIO. Addressing style is auto-selected (path-style for non-AWS endpoints).

• Access key / Secret access key — the credentials. Leave both empty to use the host’s IAM instance / ECS / EKS role (the AWS default credential chain) for keyless deployments. The secret is stored encrypted in Splunk storage_passwords.

• Server-side encryption — optional: SSE-S3 (AES256) or SSE-KMS (with an optional KMS key id).

• Storage class — optional: e.g. STANDARD_IA, GLACIER, INTELLIGENT_TIERING.

Note

Connectivity is verified on save. When you create or update an S3 Export account, TrackMe runs a live probe (a bucket check plus a write/delete of a tiny test object). A save whose target fails the probe is rejected — mirroring how remote Splunk accounts are validated — so a broken account cannot be saved silently.

Use Test S3 connectivity in the modal to run that probe on demand — it verifies read, write and delete access and reports the round-trip time:

Once saved, the target appears in the list with its live connectivity status:

Step 2 — Enable S3 export for backups

S3 export is activated with the S3 Export account (enables S3 upload) dropdown in the Backup & Restore settings ([backup_restore] → s3_export_account) — the same settings section shown in Backup storage — retention and location (2.4.3). Leaving it empty disables S3 export. Once an account is selected, every subsequent backup run — scheduled, schema-upgrade or on-demand — uploads its archives automatically.

Step 3 — Verify uploads

The top of the Backup & Restore page shows the live S3 export status — the selected account, the latest run and how many archives were uploaded:

For each run, every archive (.tar.zst), its .full.meta / .light.meta sidecars and the run manifest.json are uploaded under <prefix>/<run_id>/. The per-archive upload outcome is also recorded on the archive’s metadata record — the s3_export_status / s3_export_uri / s3_export_error fields in kv_trackme_backup_archives_info.

Two REST endpoints back this feature: POST /trackme/v2/backup_and_restore/test_s3 tests connectivity on demand, and POST /trackme/v2/backup_and_restore/export_s3 uploads existing archives (unlike the fail-open auto-upload, the on-demand endpoint reports failures). See the API Reference dashboard inside TrackMe for the exact request shape.

Monitoring failures

Because S3 export is fail-open, a silent upload failure would otherwise go unnoticed. TrackMe surfaces it two ways:

• The Configuration Guardian raises an s3_export_failing alert when the configured account is unreachable (a proactive daily probe), the configuration is invalid, or the latest run failed to upload one or more archives. It self-heals once a subsequent run uploads cleanly.

• The per-archive s3_export_status field (ok / failed) and the post_backup response s3_export summary record exactly which archives uploaded and which did not.

Retention of S3 objects

TrackMe does not delete objects from the S3 bucket — S3-side retention is intentionally left to a bucket lifecycle policy. This is a deliberate security and reliability choice:

• Least privilege. TrackMe’s S3 credentials can be scoped to write-only (PutObject). The whole point of an off-site copy is to survive the primary deployment being compromised; granting delete rights on the DR copy would let a compromise of the Splunk host (or the stored secret) wipe your off-site backups too.

• Immutability. Combine a lifecycle expiration rule with Object Lock (WORM) for ransomware-resistant, immutable-until-expiry retention — something a TrackMe-driven delete could not provide.

• Reliability. Expiration runs server-side and asynchronously, with no extra failure mode in the backup path.

Scope the lifecycle rule to the TrackMe prefix so it never affects unrelated objects. For example, on MinIO:

mc ilm rule add --expire-days 30 <alias>/<bucket> --prefix "<prefix>/"

On AWS S3, configure an equivalent Lifecycle configuration with an Expiration rule filtered to the same prefix (and NoncurrentVersionExpiration if bucket versioning is enabled).

Backup archive metadata KVstore

Every archive’s metadata is recorded in the cluster-replicated kv_trackme_backup_archives_info collection. You can inspect it directly with SPL:

| inputlookup trackme_backup_archives_info | sort 0 - mtime

Notable fields (3.0.0 onwards):

• backup_run_id — groups archives produced by the same backup run.

• archive_scope — tenant or global.

• tenant_id — set when archive_scope=tenant, empty for global.

• archive_schema_version — 3.0.0 for the multi-archive format.

• archive_sha256 — checksum captured at archive creation time.

• server_name — the SHC peer that produced the archive. Used by the cross-peer delegation logic when you trigger an operation from a different peer.

• mtime — archive timestamp.

• s3_export_account / s3_export_status / s3_export_uri / s3_export_error — set when S3 export is enabled: the target account, the per-archive upload outcome (ok / failed), the resulting s3:// URI and any failure detail.

TrackMe itself does not restore the records of kv_trackme_backup_archives_info. The collection is auto-rebuilt from archive metadata, so a missing record (e.g. after an import) is recreated on the next scan.

Importing and exporting backup archives

Use Import a backup archive to move a backup file from one TrackMe instance to another — useful for migrations, clones or for replaying a customer-supplied backup in a lab. Both 3.0.0 .tar.zst (single archive) and legacy .tgz files are accepted.

Use Download a backup archive in the top section of the page to pull a specific archive out of the TrackMe deployment. The UI lets you pick the run first, then the archive within the run.

The download path uses a memory-efficient two-step token flow under the hood (POST to get a token, GET to stream the file). The UI handles this transparently. When scripting against the REST API directly, the same two-step flow applies; see the API Reference dashboard inside TrackMe for the exact endpoint shape.

Search Head Cluster considerations

In an SHC, archives physically live on the SH peer that produced them, but every operation (list, dry-run, restore, download, delete, export) can be initiated from any peer. TrackMe handles this transparently:

• Each archive’s record carries the producing peer’s server_name.

• When you trigger an operation from a peer that doesn’t own the archive, TrackMe delegates the work to the owning peer via an internal REST call.

• Listing and grouping work on any peer because kv_trackme_backup_archives_info is cluster-replicated.

You don’t need to know which peer owns which archive — the UI works the same way regardless.

Legacy archive compatibility (pre-2.3.22)

Archives produced by TrackMe 2.3.21 and earlier (single-archive .tgz or .tar.zst with archive_schema_version 1.0.0 / 2.0.0) remain fully restorable indefinitely:

• They appear in the UI under a clearly-labelled Legacy archives grouping.

• The legacy restore code path is preserved unchanged — restoring an old archive uses the same logic as a 2.3.21 deployment would have done.

• Import accepts legacy .tgz files transparently.

• trackmecheckbackups recognises both legacy and new filenames.

Recommended after upgrade: trigger a fresh backup once on 2.3.23 to gain per-tenant restore granularity for the new run. Existing legacy archives remain available for disaster recovery from cold storage.

Logs and troubleshooting

All backup and restore operations are logged under trackme:rest_api:

index=_internal sourcetype=trackme:rest_api source="*trackme_rest_handler_backup_and_restore.py"

For a failed restore, the Full response JSON block at the bottom of the restore modal carries the per-step structured response — that’s the first place to look. The same structured payload is also written to the restore job record in kv_trackme_backup_restore_jobs and can be retrieved later by job ID even after the modal is closed.