.. _trackme_admin_cicd: TrackMe CI/CD management (TCM) ############################## .. admonition:: Minimal version required: 2.0.36 - The following documentation and capabilities require version 2.0.36 and later - It also implies an additional package called TCM, which stands for the TrackMe Configuration Manager - This package is provided to all TrackMe users at the following URL: https://downloads.trackme-solutions.com/TA-trackme-conf-manager .. image:: img_v2/tcm/diagram.png :alt: diagram.png :align: center :width: 1000px :class: with-border Introduction to the TrackMe Configuration Manager ================================================= In some contexts, it is not allowed to perform any kind of configuration in a Production environment context, and all configuration need to be performed first in a non Production environment, published to a versioning source control and finally released in a strict manner to the target environment(s). TrackMe is a highly flexible and dynamic solution which implies on the fly creation of knowledge objects and associated content, generally stored in KVstores. Therefore, typical approaches that work for more basic applications cannot be transposed to complex applications such as TrackMe, to achieve these goals we provide a configuration manager tool called TrackMe Configuration Manager. (TCM) Downloading TCM =============== **The TCM package is not available on Splunk Base, it can be freely downloaded at the following URL on our download Website:** - https://downloads.trackme-solutions.com/TA-trackme-conf-manager We do not publish this package in Splunk Base on purpose, notably to allow Splunk Cloud customers to repackage TCM (as described in the next sections) and publish it to Splunk Cloud easily. Principles ========== **The TCM workflow relies on the following logic:** **In non Production:** - In the non Production environment, the TCM is deployed and running in **receiver** mode - TrackMe is configured via a builtin option to enable sending **transactions** to the TCM - When an admin level operation is made in non Production, such as creating a new Virtual Tenant, the relevant transaction is written to the TCM JSON journal file - The TCM journal is then taken in charge by your CI/CD pipeline, packaged with your TCM release and published to your versioning control **In Production:** - The TCM is deployed and configured in **producer** mode - The builtin TCM scheduled backend is enabled - When the Python backend is executed, **transactions** from the TCM journal are read and verified - If the transaction has not been processed yet, TCM runs the associated operation by performing the exact same REST API calls to TrackMe that were made in non Production - Once the transaction has been performed, a record is written to the TCM resilient KVstore, transactions are uniquely associated with a transaction MD5 - At the next iteration, TCM knows that the transaction is done already, and does not need to be processed again Configuring TrackMe and TCM in non Production ============================================= Note about SHC in non Production -------------------------------- - Transactions are written a local file in the TCM application, which would not be replicated amongst the members of the SHC - Therefore, the TCM can be used in non Production in a SHC cluster, however you will need to manage the synchronisation of the journal file - Ideally, you should run TCM in a standalone non Production environment to avoid having to deal with this question Non Production workflow ----------------------- In non Production, which means where the configuration of TrackMe is made, the TCM is deployed in its default configuration, which acts in **receiver** mode by default. **TrackMe needs to be instructed to send the transactions to TCM:** .. image:: img_v2/tcm/screen001.png :alt: screen1.png :align: center :width: 1200px :class: with-border **Which corresponds to the following option in TrackMe configuration file:** :: [trackme_general] # Enable the TrackMe Config Manager (TCM) and sets the mode, the TCM is used for CI/CD purposes, the receiver mode sends the transaction to TCM, the replay mode replays transactions from TCM enable_conf_manager_receiver = 0 **By enabling this option, when TrackMe performs an admin level operation which creates and manages contents such as Virtual Tenants, TrackMe performs a REST call to the TCM API:** :: https:///services/trackme_conf_manager/v1/conf_manager_receiver **The TCM API receives the following information:** - ``transaction_request``: The request payload for the TrackMe API call, these are all the instructions underneath a REST call made to TrackMe's API endpoints - ``transaction_http_mode``: The requested HTTP mode for the transaction (GET, POST, DELETE) - ``transaction_http_service``: The TrackMe API service that was requested for this transaction, this is the TrackMe API endpoint that was contacted for this operation **TrackMe REST API logs the TCM forwarding actions:** *success:* :: trackme_send_to_tcm was successfully executed *Example:* :: index=_internal sourcetype=trackme:rest_api trackme_send_to_tcm :: 2023-06-05 21:55:38,258 INFO trackme_rest_handler_splk_replica_trackers_admin.py post_replica_tracker_create 224 trackme_send_to_tcm was successfully executed *In case of failures:* :: trackme_send_to_tcm has failed with exception= **On the file-system, the TCM writes transactions to a JSON journal file in local:** :: /opt/splunk/etc/apps/TA-trackme-conf-manager/local/trackme_conf_manager_transactions_journal.json **Each transaction is a JSON object which represents the REST call to be performed depending on the actions requested in TrackMe, through the UI and the REST API:** *Example:* :: "9c2f7f8264a86913f34a97e161b128e8": { "transaction_request": { "tenant_desc": "SIEM", "tenant_name": "feeds-tracking", "tenant_roles_admin": [ "trackme_admin" ], "tenant_roles_power": [ "trackme_power" ], "tenant_roles_user": [ "trackme_user" ], "tenant_owner": "svc-trackme", "tenant_idx_settings": "{\"trackme_summary_idx\": \"trackme_summary\", \"trackme_audit_idx\": \"trackme_audit\", \"trackme_notable_idx\": \"trackme_notable\", \"trackme_metric_idx\": \"trackme_metrics\"}", "tenant_dsm_enabled": true, "tenant_dsm_sampling_obfuscation": "disabled", "tenant_dhm_enabled": true, "tenant_dhm_alerting_policy": "track_per_host", "tenant_mhm_enabled": true }, "transaction_http_mode": "post", "transaction_http_service": "/services/trackme/v2/vtenants/admin/add_tenant", "ctime": 1686000734.0977793 } A transaction is identified by a unique MD5 hash, this same hash will be used by TCM on the producer side (therefore in Production) to identify transactions which have been processed already, and transactions which are pending. Configuring TrackMe and TCM in Production ========================================= Note about SHC in Production ---------------------------- - SHC is fully supported for all TCM replay transactions TCM packaging and deployment ---------------------------- **In Production, therefore the target environment where transactions will be replayed, the TCM must be deployed in Producer mode:** *Through the UI:* .. image:: img_v2/tcm/screen002.png :alt: screen002.png :align: center :width: 1200px :class: with-border *Which corresponds to the following configuration:* *ta_trackme_conf_manager_settings.conf* :: [trackme_general] conf_manager_role = producer **TCM packaging:** *Depending on your context:* - If you are a **Splunk Cloud** customer, you need to include the journal file in the default directory, package the application release and deploy it through Splunk ACS (through the API or UI), once Appinspect is passed, TCM is ready - If you are a **Splunk Enterprise** customer, you can equally publish the journal in default or local, but local has precedence over the default directory TrackMe configuration --------------------- TrackMe does not require any configuration for TCM. TCM acts independently once it is deployed and configured, and will interact with TrackMe as needed to replay the transactions and apply your settings accordingly. However, you can disable all admin levels operations through the UI via the following settings (to avoid allowing any admin to generate a conflict between changes) and remove any related settings from TrackMe's user interfaces: .. image:: img_v2/tcm/screen003.png :alt: screen003.png :align: center :width: 1200px :class: with-border As well, note that forwarding to TCM must be disabled in the Production context. (which is the default) Viewing the transaction journal ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **You can access the transaction journal via the following command:** :: | trackmeconfmanager mode=get *Example:* .. image:: img_v2/tcm/screen004.png :alt: screen004.png :align: center :width: 1200px :class: with-border Checking transactions statuses ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **You can check the status of transactions via the following command:** :: | trackmeconfmanager mode=simulation A transaction which has not been completed yet a ``transaction_info`` message explaining the action which will be taken. A transaction which has been completed already shows a ``transaction_info`` message: "This transaction has been processed already, the Conf Manager will not attempt to process it again". *In the following example, all transactions have been proceeded:* .. image:: img_v2/tcm/screen005.png :alt: screen005.png :align: center :width: 1200px :class: with-border *In the following example, transactions are pending and would be proceeded:* .. image:: img_v2/tcm/screen006.png :alt: screen006.png :align: center :width: 1200px :class: with-border Enabling the scheduled report trackme_conf_manager_producer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **The TCM packages provides a builtin scheduled report which is disabled by default, the report runs the following command:** :: | trackmeconfmanager mode=live This is the process which replays transactions and configures TrackMe according to the transaction journal provided by your CI/CD pipeline. **The process works as following:** - For each transaction, the Python backend verifies if the transaction can be found in the resilient KVstore ``kv_trackme_conf_manager``: :: | inputlookup trackme_conf_manager | eval key=_key - If the transaction has not been proceeded, TCM performs the associated call to the TrackMe REST API - If the transaction has been proceeded, then there are no actions to be taken *The TCM transaction replay backend logs its activity into the following logs:* :: index=_internal sourcetype=trackme_conf_manager:trackmeconfmanager Troubleshooting =============== TCM receiver REST API --------------------- **When a transaction is sent from the non Production environment to TCM, activity is logged here:** :: index=_internal sourcetype=trackme_conf_manager:rest_api *example:* :: 2023-06-05 21:54:12,145 INFO trackme_conf_manager_rest_handler.py post_conf_manager_receiver 216 conf manager received configuration, resp_dict="{'transaction_request': {'tenant_id': 'replica-demo', 'source_tenant_id': 'cim-demo', 'component': 'cim', 'root_constraint': 'object=*', 'update_comment': 'No comment for update.', 'owner': 'svc-trackme'}, 'transaction_http_mode': 'post', 'transaction_http_service': '/services/trackme/v2/splk_replica_trackers/admin/replica_tracker_create'}" 2023-06-05 21:54:12,158 INFO trackme_conf_manager_rest_handler.py post_conf_manager_receiver 284 {'response': 'transaction received with md5=5df36dfae06b237408a0cecd30e87015 and successfully written to the journal', 'journal': '/opt/splunk/etc/apps/TA-trackme-conf-manager/local/trackme_conf_manager_transactions_journal.json'} TCM transaction replay (command trackmeconfmanager) --------------------------------------------------- **Activity of the trackmeconfmanager command is logged here:** :: index=_internal sourcetype=trackme_conf_manager:trackmeconfmanager *example:* :: 2023-06-06 09:16:29,916 INFO trackmeconfmanager.py generate 156 verifying transaction status, transaction_md5="ca315b5f58e2766e60bf231e86a5f551" 2023-06-06 09:16:29,916 INFO trackmeconfmanager.py generate 162 transaction was not found in the KVstore and will be processed in live mode 2023-06-06 09:16:41,030 INFO trackmeconfmanager.py generate 219 transaction was registered successfully to the KVstore with key="ca315b5f58e2766e60bf231e86a5f551"