TrackMe CI/CD management (TCM)

Minimal version required: 2.0.36

diagram.png

Introduction to the TrackMe Configuration Manager

In some contexts, it is not allowed to perform any kind of configuration in a Production environment context, and all configuration need to be performed first in a non Production environment, published to a versioning source control and finally released in a strict manner to the target environment(s).

TrackMe is a highly flexible and dynamic solution which implies on the fly creation of knowledge objects and associated content, generally stored in KVstores.

Therefore, typical approaches that work for more basic applications cannot be transposed to complex applications such as TrackMe, to achieve these goals we provide a configuration manager tool called TrackMe Configuration Manager. (TCM)

Downloading TCM

The TCM package is not available on Splunk Base, it can be freely downloaded at the following URL on our download Website:

We do not publish this package in Splunk Base on purpose, notably to allow Splunk Cloud customers to repackage TCM (as described in the next sections) and publish it to Splunk Cloud easily.

Principles

The TCM workflow relies on the following logic:

In non Production:

  • In the non Production environment, the TCM is deployed and running in receiver mode

  • TrackMe is configured via a builtin option to enable sending transactions to the TCM

  • When an admin level operation is made in non Production, such as creating a new Virtual Tenant, the relevant transaction is written to the TCM JSON journal file

  • The TCM journal is then taken in charge by your CI/CD pipeline, packaged with your TCM release and published to your versioning control

In Production:

  • The TCM is deployed and configured in producer mode

  • The builtin TCM scheduled backend is enabled

  • When the Python backend is executed, transactions from the TCM journal are read and verified

  • If the transaction has not been processed yet, TCM runs the associated operation by performing the exact same REST API calls to TrackMe that were made in non Production

  • Once the transaction has been performed, a record is written to the TCM resilient KVstore, transactions are uniquely associated with a transaction MD5

  • At the next iteration, TCM knows that the transaction is done already, and does not need to be processed again

Configuring TrackMe and TCM in non Production

Note about SHC in non Production

  • Transactions are written a local file in the TCM application, which would not be replicated amongst the members of the SHC

  • Therefore, the TCM can be used in non Production in a SHC cluster, however you will need to manage the synchronisation of the journal file

  • Ideally, you should run TCM in a standalone non Production environment to avoid having to deal with this question

Non Production workflow

In non Production, which means where the configuration of TrackMe is made, the TCM is deployed in its default configuration, which acts in receiver mode by default.

TrackMe needs to be instructed to send the transactions to TCM:

screen1.png

Which corresponds to the following option in TrackMe configuration file:

[trackme_general]
# Enable the TrackMe Config Manager (TCM) and sets the mode, the TCM is used for CI/CD purposes, the receiver mode sends the transaction to TCM, the replay mode replays transactions from TCM
enable_conf_manager_receiver = 0

By enabling this option, when TrackMe performs an admin level operation which creates and manages contents such as Virtual Tenants, TrackMe performs a REST call to the TCM API:

https://<local splunkd uri>/services/trackme_conf_manager/v1/conf_manager_receiver

The TCM API receives the following information:

  • transaction_request: The request payload for the TrackMe API call, these are all the instructions underneath a REST call made to TrackMe’s API endpoints

  • transaction_http_mode: The requested HTTP mode for the transaction (GET, POST, DELETE)

  • transaction_http_service: The TrackMe API service that was requested for this transaction, this is the TrackMe API endpoint that was contacted for this operation

TrackMe REST API logs the TCM forwarding actions:

success:

trackme_send_to_tcm was successfully executed

Example:

index=_internal sourcetype=trackme:rest_api trackme_send_to_tcm

2023-06-05 21:55:38,258 INFO trackme_rest_handler_splk_replica_trackers_admin.py post_replica_tracker_create 224 trackme_send_to_tcm was successfully executed

In case of failures:

trackme_send_to_tcm has failed with exception=<exception>

On the file-system, the TCM writes transactions to a JSON journal file in local:

/opt/splunk/etc/apps/TA-trackme-conf-manager/local/trackme_conf_manager_transactions_journal.json

Each transaction is a JSON object which represents the REST call to be performed depending on the actions requested in TrackMe, through the UI and the REST API:

Example:

"9c2f7f8264a86913f34a97e161b128e8": {
    "transaction_request": {
        "tenant_desc": "SIEM",
        "tenant_name": "feeds-tracking",
        "tenant_roles_admin": [
            "trackme_admin"
        ],
        "tenant_roles_power": [
            "trackme_power"
        ],
        "tenant_roles_user": [
            "trackme_user"
        ],
        "tenant_owner": "svc-trackme",
        "tenant_idx_settings": "{\"trackme_summary_idx\": \"trackme_summary\", \"trackme_audit_idx\": \"trackme_audit\", \"trackme_notable_idx\": \"trackme_notable\", \"trackme_metric_idx\": \"trackme_metrics\"}",
        "tenant_dsm_enabled": true,
        "tenant_dsm_sampling_obfuscation": "disabled",
        "tenant_dhm_enabled": true,
        "tenant_dhm_alerting_policy": "track_per_host",
        "tenant_mhm_enabled": true
    },
    "transaction_http_mode": "post",
    "transaction_http_service": "/services/trackme/v2/vtenants/admin/add_tenant",
    "ctime": 1686000734.0977793
}

A transaction is identified by a unique MD5 hash, this same hash will be used by TCM on the producer side (therefore in Production) to identify transactions which have been processed already, and transactions which are pending.

Configuring TrackMe and TCM in Production

Note about SHC in Production

  • SHC is fully supported for all TCM replay transactions

TCM packaging and deployment

In Production, therefore the target environment where transactions will be replayed, the TCM must be deployed in Producer mode:

Through the UI:

screen002.png

Which corresponds to the following configuration:

ta_trackme_conf_manager_settings.conf

[trackme_general]
conf_manager_role = producer

TCM packaging:

Depending on your context:

  • If you are a Splunk Cloud customer, you need to include the journal file in the default directory, package the application release and deploy it through Splunk ACS (through the API or UI), once Appinspect is passed, TCM is ready

  • If you are a Splunk Enterprise customer, you can equally publish the journal in default or local, but local has precedence over the default directory

TrackMe configuration

TrackMe does not require any configuration for TCM.

TCM acts independently once it is deployed and configured, and will interact with TrackMe as needed to replay the transactions and apply your settings accordingly.

However, you can disable all admin levels operations through the UI via the following settings (to avoid allowing any admin to generate a conflict between changes) and remove any related settings from TrackMe’s user interfaces:

screen003.png

As well, note that forwarding to TCM must be disabled in the Production context. (which is the default)

Viewing the transaction journal

You can access the transaction journal via the following command:

| trackmeconfmanager mode=get

Example:

screen004.png

Checking transactions statuses

You can check the status of transactions via the following command:

| trackmeconfmanager mode=simulation

A transaction which has not been completed yet a transaction_info message explaining the action which will be taken.

A transaction which has been completed already shows a transaction_info message: “This transaction has been processed already, the Conf Manager will not attempt to process it again”.

In the following example, all transactions have been proceeded:

screen005.png

In the following example, transactions are pending and would be proceeded:

screen006.png

Enabling the scheduled report trackme_conf_manager_producer

The TCM packages provides a builtin scheduled report which is disabled by default, the report runs the following command:

| trackmeconfmanager mode=live

This is the process which replays transactions and configures TrackMe according to the transaction journal provided by your CI/CD pipeline.

The process works as following:

  • For each transaction, the Python backend verifies if the transaction can be found in the resilient KVstore kv_trackme_conf_manager:

| inputlookup trackme_conf_manager | eval key=_key

  • If the transaction has not been proceeded, TCM performs the associated call to the TrackMe REST API

  • If the transaction has been proceeded, then there are no actions to be taken

The TCM transaction replay backend logs its activity into the following logs:

index=_internal sourcetype=trackme_conf_manager:trackmeconfmanager

Troubleshooting

TCM receiver REST API

When a transaction is sent from the non Production environment to TCM, activity is logged here:

index=_internal sourcetype=trackme_conf_manager:rest_api

example:

2023-06-05 21:54:12,145 INFO trackme_conf_manager_rest_handler.py post_conf_manager_receiver 216 conf manager received configuration, resp_dict="{'transaction_request': {'tenant_id': 'replica-demo', 'source_tenant_id': 'cim-demo', 'component': 'cim', 'root_constraint': 'object=*', 'update_comment': 'No comment for update.', 'owner': 'svc-trackme'}, 'transaction_http_mode': 'post', 'transaction_http_service': '/services/trackme/v2/splk_replica_trackers/admin/replica_tracker_create'}"
2023-06-05 21:54:12,158 INFO trackme_conf_manager_rest_handler.py post_conf_manager_receiver 284 {'response': 'transaction received with md5=5df36dfae06b237408a0cecd30e87015 and successfully written to the journal', 'journal': '/opt/splunk/etc/apps/TA-trackme-conf-manager/local/trackme_conf_manager_transactions_journal.json'}

TCM transaction replay (command trackmeconfmanager)

Activity of the trackmeconfmanager command is logged here:

index=_internal sourcetype=trackme_conf_manager:trackmeconfmanager

example:

2023-06-06 09:16:29,916 INFO trackmeconfmanager.py generate 156 verifying transaction status, transaction_md5="ca315b5f58e2766e60bf231e86a5f551"
2023-06-06 09:16:29,916 INFO trackmeconfmanager.py generate 162 transaction was not found in the KVstore and will be processed in live mode
2023-06-06 09:16:41,030 INFO trackmeconfmanager.py generate 219 transaction was registered successfully to the KVstore with key="ca315b5f58e2766e60bf231e86a5f551"