Backing up and Restoring TrackMe

Hint

TrackMe Backup and Restore new features since 2.1.5: backup, restore, clone and migrate TrackMe Knowledge Objects & KVstore collections

Since TrackMe 2.1.5, a new set of features have been introduced to allow for more flexibility and capabilities in managing TrackMe Knowledge Objects and KVstore collections.
TrackMe now allows for the backup, restore, clone and migrate of TrackMe Knowledge Objects and KVstore collections records, entirely within TrackMe.
These features leverage new TrackMe endpoints and utilities, and a refreshed dashboard allowing to easily access to command examples and reviewing available backups.
These new capabilities leverage the Splunk API regarding backing up and restoring Knowledge Objects, and work whenever you are a Splunk Enterprise or Cloud customer, in a standalone or Search Head Cluster context.
This new sets of incredibly powerful and flexible features allow TrackMe to be entirely autonomous regarding its backup and restore capabilities, and to be able to manage its own Knowledge Objects and KVstore collections records in a very granular way.
This documentation has been updated to reflect these new capabilities, and to provide guidance on how to use them.

Note

Limitations:

Disabled Virtual Tenants are not backed up, and therefore not restorables, when performing a restore operation, TrackMe will automatically force purge a Virtual Tenant if it was disabled when the backup was performed.
Alert actions restoration: TrackMe can restore any alert that was created with TrackMe, however only TrackMe built-in alerts actions and Splunk Email alert actions are restored. (external alert actions will need to be re-configured manually once restored)

Introduction to backup and restore

TrackMe is a complex Splunk application, backing up and restoring means that the following need to be taken into account:

In TrackMe, Splunk Knowledge Objects are created on the fly, this includes KVstore definitions, transforms, macros, reports and alerts, etc…
TrackMe heavily relies on KVstores, the application stores various states as well as its own data and configuration in KVstores.
When backing up and restoring TrackMe, both aspects have to be managed together, you cannot restore TrackMe configuration files without restoring the KVstore data and vice versa.
TrackMe provides builtin advanced capabilities to backup and restore its Knowledge Objects and resilient data stores. (KVstores)

Backing up TrackMe

Automated backups in TrackMe

TrackMe automatically performs backups on the following circumstances:

Every days, the scheduled report trackme_backup_scheduler executes during the night and performs a full backup of TrackMe.
When TrackMe is upgraded to a new release, TrackMe performs a backup when the first Virtual Tenant is upgraded through a process called schema upgrade.

TrackMe backups location:

TrackMe backups are stored in the backup directory at the root of the TrackMe application directory.
The backup directory is located at: $SPLUNK_HOME/etc/apps/trackme/backup/
The format of the files are compressed tarballs, with the following naming convention: trackme-backup-YYYYMMDD-HHMMSS.tgz

Reviewing TrackMe backups

To review the available backups, you can use the builtin dashboard “TrackMe Backup and Restore” available in the menu API &amp Tooling:

The first part of the dashboard shows various use cases and command examples:

Scroll down to see available backups in this TrackMe deployments, or access logs:

TrackMe logs the backup context in the comment metadata, if the backup is executed during the context of the release upgrade, the comment mentions this and the schema version upgrade information.

Checking, Inspecting and Reviewing TrackMe backups

The most practical way to easily check, inspect and review TrackMe’s backups, including Knowledge Objects and their definition, is to use a built-in command called trackmecheckbackups:

| trackmecheckbackups archives_list="*"

What this backend does:

Lists and iterates over all available TrackMe backups.
Verifies that the backup archives can successfully be extracted.
Access and iterate through the archives metadata.
Render the archives contents and metadata in pure SPL and on a per archive basis.

Notes:

The command does not give access to the KVstore records themselves, but gives detailed metadata for each kvstore collection. (number of records, etc)
Knowledge Objects can be fully accessed, and you can for instance review the definition of a given search, alert or macro at some points in time via the archive.

Accessing the Backup metadata KVstore

TrackMe stores the metadata of the backups in a dedicated KVstore collection, you can access this collection using the following SPL command:

| inputlookup trackme_backup_archives_info | sort - limit=0 mtime

The KVstore is used in the dashboard to provide high-level information about the backups, such as the number of archives, the size of the archives, the creation date, etc… This KVstore is also used by the schema upgrade process to determine the latest backup available.

Finally, note that TrackMe does not restore the KVstore records for this specific KVstore, the reason is that TrackMe automatically discovers available backups, and will re-create any missing records for a backup which would not be currently known to the system.

Anatomy of a TrackMe backup

A TrackMe backup archive contains the following elements:

The backup archive itself, a compressed tarball containing all the files and directories of the backup.
A full metadata file, containing the metadata of the backup, such as the creation date, the size of the archive, etc…
A light metadata file, containing a summary of the backup, such as the number of Knowledge Objects, the number of KVstore collections, etc…

Light metadata content example:

{
"trackme_version": "2.1.5",
"backup_archive": "/opt/splunk/etc/apps/trackme/backup/trackme-backup-20241123-020021.tgz",
"server_name": "xxxxxxxxxxx",
"comment": "Backup initiated by splunk-system-user, date: Sat Nov 23 02:00:21 2024",
"kvstore_collections_not_empty": "199",
"kvstore_collections_empty": "148",
"knowledge_objects_summary": {
        "tenants_enabled_count": 9,
        "total": 522,
        "kvstore_collections": 191,
        "transforms": 191,
        "reports": 120,
        "alerts": 9,
        "macros": 11
    }
}

Example:

cd $SPLUNK_HOME/etc/apps/trackme/backup/

-rw------- 1 splunk splunk 713476 Nov 22 22:36 trackme-backup-20241122-223558.tgz
-rw------- 1 splunk splunk  45149 Nov 22 22:36 trackme-backup-20241122-223558.tgz.full.meta
-rw------- 1 splunk splunk    518 Nov 22 22:36 trackme-backup-20241122-223558.tgz.light.meta

The content of the archive itself:

One JSON file per KVstore collection, containing the records of the collection. (Note: the file will be empty if the collection is empty)
One JSON file per Virtual Tenant containing the Knowledge Objects of the Virtual Tenant.
One JSON file per Virtual Tenant containing the Virtual Tenant account information.

kv_<kvstore_transform_reference>.json
... <repeated for each KVstore>
tenant_<tenant_id>_knowledge_objects.json
tenant_<tenant_id>_vtenant_account.json

Restoring TrackMe

Restoring TrackMe can be achieved with flexibility, and different options are available depending on your needs:

full restoration: You can entirely restore TrackMe Knowledge Objects and KVstore records from a backup.
full restoration targeting all or a list of Virtual Tenants, without deleting and replacing existing Knowledge Objects: You can restore TrackMe Knowledge Objects and KVstore records from a backup, without altering existing Knowledge Objects.
full restoration targeting all or a list of Virtual Tenants, with or without replacing Knowledge Objects, and without restoring KVstore records: You can restore TrackMe Knowledge Objects from a backup, without restoring KVstore records.
partial restoration of Knowledge Objects: You can restore a specific list of Knowledge Objects from a backup.
partial restoration of KVstore collections records: You can restore a specific list of KVstore collections and their records from a backup.
mixed restoration: You can mix and match the above options to achieve a specific restoration scenario.

The next steps will guide you through the restoration process based on these different scenarios.

Restore endpoint API and its options

The TrackMe API restore endpoint handle these operations, you can review backup and restore endpoints and their usage in the API Reference dashboard:

You can also run:

| trackmeapiautodocs target="endpoints" | search resource_group="backup_and_restore"

Endpoint options:

backup_archive: The archive file to be restored, the tarball compressed file must be located in the backup directory of the trackMe application.
dry_run: (true / false) OPTIONAL: if true, the endpoint will only verify that the archive can be found and successfully extracted, there will be no modifications at all. (default to true)
knowledge_objects_lists: (all / comma separated list of objects to be restored) OPTIONAL: restore all knowledge objects that were backed up in the submitted archive, or provide a comma separated list of objects to be restored (defaults to all) - This requires a backup archive generated with TrackMe 2.1.5 or later
knowledge_objects_replace_existing: (true / false) OPTIONAL: if the object already exists, it will be replaced (default to true) - This requires a backup archive generated with TrackMe 2.1.5 or later
knowledge_objects_tenants_scope: (all / comma separated list of tenant_id) OPTIONAL: restore the knowledge objects for all tenants or provide a comma separated list of tenant_id to be restored (defaults to all) - This requires a backup archive generated with TrackMe 2.1.5 or later
kvstore_collections_clean_empty: (true / false) OPTIONAL: if the collection was empty in the backup, restoring will empty any existing record in the collection (default to true)
kvstore_collections_restore_non_tenants_collections: (true / false) OPTIONAL: restore non-tenants collections (default to true), non tenants collections are KVstore collections which are tenant specific, such as user preferences, user settings, etc. If set to True, the content of the collection content will be restored from the backup.
kvstore_collections_scope: (all / comma separated list of KVstore collections) OPTIONAL: restore all collections that were backed up in the submitted archive, or provide a comma separated list of collections to be restored (defaults to all)
restore_knowledge_objects: (true / false) OPTIONAL: restore the knowledge objects from the submitted archive (default to true) - This requires a backup archive generated with TrackMe 2.1.5 or later
restore_kvstore_collections: (true / false) OPTIONAL: restore the KVstore collections from the submitted archive (default to true)
restore_virtual_tenant_accounts: (true / false) OPTIONAL: check and restore the virtual tenant accounts from the submitted archive (default to true)
restore_virtual_tenant_main_kvrecord: (true / false) OPTIONAL: check and restore the virtual tenant main record from the submitted archive (default to true)

Restore operations examples

Full restoration of TrackMe

Example:

| trackme
url="/services/trackme/v2/backup_and_restore/restore" mode="post" body="{'backup_archive': '/opt/splunk/etc/apps/trackme/backup/trackme-backup-20241117-220628.tgz', 'dry_run': 'false'}"

This example above will:

For each Virtual Tenant, restore the Virtual Tenant account
For each Virtual Tenant, restore the central KVstore record
For each Virtual Tenant, restore Knowledge Objects, for each Knowledge Object, if the Knowledge Object exists already, it will be deleted and re-created from the backup
For each Virtual Tenant KVstore collection, restore the KVstore collection records (if the collection was empty in the backup, the collection will be emptied)
Restore central and non-tenants related KVstore collections records

Restore example: restore a specific Virtual Tenant, including all Virtual Tenants Knowledge Objects and all TrackMe KVstore records for this Virtual Tenant only

| trackme
      url="/services/trackme/v2/backup_and_restore/restore" mode="post" body="{'backup_archive': '/opt/splunk/etc/apps/trackme/backup/trackme-backup-20241117-220628.tgz', 'dry_run': 'false', 'knowledge_objects_tenants_scope': 'my_tenant', 'knowledge_objects_replace_existing': 'true', 'kvstore_collections_clean_empty': 'true', 'kvstore_collections_restore_non_tenants_collections': 'false'}"

This example above will:

For the Virtual Tenant “my_tenant”, restore the Virtual Tenant account
For the Virtual Tenant “my_tenant”, restore the central KVstore record
For the Virtual Tenant “my_tenant”, restore Knowledge Objects, for each Knowledge Object, if the Knowledge Object exists already, it will be deleted and re-created from the backup

Restore example: restore one or more specific Knowledge Objects, and nothing else (if the report calls itself another report, the parent report will be also restored automatically)

| trackme
url="/services/trackme/v2/backup_and_restore/restore" mode="post" body="{'backup_archive': '/opt/splunk/etc/apps/trackme/backup/trackme-backup-20241117-220628.tgz', 'dry_run': 'false', 'knowledge_objects_lists': 'my-tracker-001,my-tracker-002', 'restore_virtual_tenant_accounts': 'false', 'restore_virtual_tenant_main_kvrecord': 'false', 'knowledge_objects_replace_existing': 'true', 'restore_kvstore_collections': 'false'}"

This example above will:

Restore the Knowledge Objects “my-tracker-001” and “my-tracker-002” from the backup
If the Knowledge Object exists already, it will be deleted and re-created from the backup
No Virtual Tenant account will be restored
No Virtual Tenant KVstore record will be restored
No KVstore collection records will be restored

Restore example: restore one or more specific KVstore collections contents, and nothing else (kvstore_collections_clean_empty allows controlling if the collection should be empty, if it was actually empty during the backup)

| trackme
url="/services/trackme/v2/backup_and_restore/restore" mode="post" body="{'backup_archive': '/opt/splunk/etc/apps/trackme/backup/trackme-backup-20241117-220628.tgz', 'dry_run': 'false', 'kvstore_collections_scope': 'kv_trackme_dsm_tenant_mytenant,kv_trackme_dsm_priority_policies_tenant_test-restore', 'kvstore_collections_clean_empty': 'true', 'restore_knowledge_objects': 'false', 'restore_virtual_tenant_accounts': 'false', 'restore_virtual_tenant_main_kvrecord': 'false'}"

This example above will:

Restore the KVstore collections “kv_trackme_dsm_tenant_mytenant” and “kv_trackme_dsm_priority_policies_tenant_test-restore” from the backup
If the collection was empty in the backup, the collection will be emptied
No Knowledge Objects will be restored
No Virtual Tenant account will be restored
No Virtual Tenant KVstore record will be restored

Exporting and Importing TrackMe backups, for migration or cloning purposes

TrackMe provides importing and exporting capabilities that can be used to migrate or clone TrackMe configurations and data between different TrackMe deployments:

Exporting a TrackMe backup (source system)

You can use the export REST API to export a TrackMe backup archive as a base64 string, which can then be used to call the import endpoint to a different Splunk instance running TrackMe (use cases: migration, cloning…) - This example shows the usage with curl

curl -k -u admin https://mysplunk-source:8089/services/trackme/v2/backup_and_restore/export_backup -X "POST" -d "{\"archive_name\": \"trackme-backup-20241120-064752.tgz\"}" --output export.json

Review the resulting file export.json.*

Importing a TrackMe backup (target system)

Then, you can use the import REST API to import a TrackMe backup archive from the base64 string, TrackMe automatically verifies the archive, if the archive is valid, it will be discovered and integrated into the available archives immediately - This example shows the usage with curl

curl -k -u admin https://mysplunk-target:8089/services/trackme/v2/backup_and_restore/import_backup -X "POST" -H "Content-Type: application/json" --data @export.json

Expected response:

{"success":"Backup imported successfully"}

Note:

TrackMe will automatically identify the backup file on the target system, and will include it in the available backups.
After an import, if you open the TrackMe Backup and Restore dashboard, you will see the imported backup in the list of available backups.

Migrating or cloning the TrackMe deployment

Once you have imported the backup on the target system, you can then proceed with the restoration process as described in the previous section, to restore the Knowledge Objects and KVstore records.

| trackme
url="/services/trackme/v2/backup_and_restore/restore" mode="post" body="{'backup_archive': '/opt/splunk/etc/apps/trackme/backup/trackme-backup-20241117-220628.tgz', 'dry_run': 'false'}"

Splunk Search Head Cluster specific considerations

In a Splunk Search Head context, especially in Splunk Cloud where you cannot access the filesystem, you may need to consider the following:

The backup file on the file-system is Search Head member specific, backup files are not replicated to other members.
This means that when calling a restore operation, you need to make sure that either the backup file is available on the member commands are executed, or that you can target the specific member of the Search Head cluster where the backup file is hosted.

Identifying the backup file member location:

TrackMe stores the information about the server name where the tgz backup file was generated.
You can see this information in the Backup and Restore dashboard under server_name.

Therefore:

You can use the export endpoint to export the backup file and by targetiing the SHC member where it is located.
Then use the import endpoint to import the backup file on the target SHC member.
Finally, you can execute the restore operation on the target SHC member.
These steps can be used whenever you need to restore in the same environment, or if the objective is to migrate or clone the TrackMe deployment from a SHC to another. (same or different Splunk deployment)

Hint

Accessing Splunk Cloud specific Search heads API*

In Splunk Cloud, you can access the Search Head members API using the following URL: https://<search_head_member_name>.<cloud stack name>.splunkcloud.com:8089
To get the fully qualified name of the server you can connected to, in Splunk Web, click on top right “Support and Services”, “About”
You can easily found out the list of Search Head members in the Cloud monitoring console, or by looking splunkd internal logs.
Finally, remember that you must allow public IP network where your connection starts from in the Splunk Cloud Splunk REST API allow list in Splunk Cloud.

Examples:

Step 1 Export from the source SHC member

curl -k -u admin https://<shc member source>:8089/services/trackme/v2/backup_and_restore/export_backup -X "POST" -d "{\"archive_name\": \"trackme-backup-20241120-064752.tgz\"}" --output export.json

Step 2: Import to the SHC member target

curl -k -u admin https://<shc member target>:8089/services/trackme/v2/backup_and_restore/import_backup -X "POST" -H "Content-Type: application/json" --data @export.json

Step 3: Restore on the SHC member target

curl -k -u admin https://<shc member target>:8089/services/trackme/v2/backup_and_restore/restore -X "POST" -H "Content-Type: application/json" -d "{\"backup_archive\": \"/opt/splunk/etc/apps/trackme/backup/trackme-backup-20241120-064752.tgz\", \"dry_run\": \"false\"}"

Accessing logs and troubleshooting

TrackMe logs all backup and restore operations in the TrackMe logs, you can access the logs using the following search:

index=_internal sourcetype=trackme:rest_api trackme_rest_handler_backup_and_restore.py