Progressive Data Contracts

About Progressive Data Contracts

  • This integration documentation provides some integration tips for TrackMe to work with Progressive Data Contracts.

  • Progressive Data Contracts were presented at Splunk .conf 2025, link: https://conf.splunk.com/watch/conf-online.html?search.event=conf and search for PLA1316.

  • Link to the Progressive Data Contracts GitHub repository: https://github.com/Progressive-Insurance/log-data-contracts

  • A data contract is a formal agreement between a service provider and a customer, detailing a data product to be exchanged. In the context of Splunk data ingestion, the exchange occurs between the “data owner” (the team or individual responsible for the originating system of Splunk events) and the Splunk Data Team (acting as data custodians). The data contract outlines the dataset description, ownership information, and the data handling requirements within Splunk.

Hint

TrackMe command trackmeyamlpath:

  • This streaming custom command was introduced in TrackMe 2.1.30.

  • The command can be used to parse YAML data, and extract automatically the fields you need.

  • Notably, it used in the example below to extract the contact field from the YAML data.

  • Example of usage: | inputlookup data-contracts | trackmeyamlpath yaml_fieldname=yaml_data

Splunk_data_contracts-architecture_and_use_cases.drawio.png

Integration in TrackMe UI: CMDB lookup search definition

TrackMe has a concept of CMDB lookup search, which allows defining a Splunk search template to retrieve and enrich knowledge about a TrackMe entity:

Assuming you have a KVstore collection and a transforms definition ready which contains the data contracts, you can easily setup a CMDB lookup search that relies on:

  • indexName: the index name of the data contract

  • sourcetypeName: the sourcetype name of the data contract

You would define the CMDB lookup search as follows: (replace the KVstore lookup name with your own)

| inputlookup data-contracts where (indexName="$data_index$" AND sourcetypeName="$data_sourcetype$")

If you want to filter out the results, you can add any Splunk Processing Language (SPL) statement, for instance:

| inputlookup data-contracts where (indexName="$data_index$" AND sourcetypeName="$data_sourcetype$") | table indexName, sourcetypeName, yaml_data

From TrackMe 2.1.30, you can also call the command trackmeyamlpath to extract fields from the YAML data if you like:

| inputlookup data-contracts where (indexName="$data_index$" AND sourcetypeName="$data_sourcetype$") | table indexName, sourcetypeName, yaml_data | trackmeyamlpath yaml_fieldname=yaml_data

Defining the CMDB lookup search in TrackMe UI:

screen001.png screen002.png

Example of result when hitting the CMDB icon:

screen003.png screen004.png

Integration in TrackMe UI: alerts triggering enrichment

The next piece of integration can be to leverage the data contracts to enrich alerts triggering.

TrackMe stateful alerts

You would add the following code to the stateful alert created by TrackMe:

Replacements:

  • Replace mytenant with your tenant name

  • Replace the name of the KVstore transforms of the data contracts

  • Adapt the fields as needed

  • Call trackmeyamlpath to extract fields from the YAML data if your KVstore contains the unparsed YAML data

screen005.png 
``` retrieve index and sourcetype from the central TrackMe collection ```
| lookup trackme_dsm_tenant_secops _key as keyid OUTPUT data_index, data_sourcetype
``` enrich from data contracts ```
| lookup data-contracts indexName as data_index, sourcetypeName as data_sourcetype OUTPUT yaml-data
| trackmeyamlpath yaml_fieldname="yaml-data"
| eval contact=if(isnull('datasets.splunkDataset.dataProductEmailDistributionName'), "support@trackme-solutions.com", 'datasets.splunkDataset.dataProductEmailDistributionName')

Then update the recipient email address so it uses the token resulting from the contact field, under the form $result.contact$:

screen006.png

TrackMe notables

For legacy alerting with TrackMe notables, the logic would be similar and would rely on enriching the notable events using the data contracts information.

Example:

index=trackme_notable tenant_id=* priority="high"
| lookup data-contracts index as indexName, sourcetype as indexSourcetype OUTPUT yaml_data
| trackmeyamlpath yaml_fieldname=yaml_data
| eval contact=if(isnull('datasets.splunkDataset.dataProductEmailDistributionName'), "support@trackme-solutions.com", 'datasets.splunkDataset.dataProductEmailDistributionName')
| table tenant_id, keyid, event_id, object, object_category, state, priority, anomaly_reason, status_message, drilldown_link, contact

Integration in TrackMe UI: managing priority or other information from the data contracts

You could also leverage further any useful information from the data contracts, to manage for instance the priority of an entity, or delay/latency thresholds.

Below is an example assuming you have a field priority in the data contracts, we would leverage the concept of TrackMe external priority management: (see: Managing priority externally)

Notes: for this example to work, we added the following fields to the data contracts, underneath Observability:

dataObservability:
  freshness:
    measureTypeCode: indexName:sourcetypeName
    latencyThresholdName: 5min
  measuredVolume:
    volumeMeasureDate: 2024-10-05
    dailyGbCount: 53
    dailyEventCount: 10120021
    avgEventByteCount: 5237
  criticity:
    priority: critical

The following search would be executed, and scheduled at your convenience, say once daily, to automatically update the external priority management:

| inputlookup trackme_dsm_tenant_secops | eval keyid=_key

``` In this example, we leverage a Splunk lookup file referencing indexes and used to define the field priority_external ```
``` enrich from data contracts ```
| lookup data-contracts indexName as data_index, sourcetypeName as data_sourcetype OUTPUT yaml-data
| trackmeyamlpath yaml_fieldname="yaml-data"

| where isnotnull('datasets.splunkDataset.dataObservability.criticity.priority') AND isnotnull('datasets.splunkDataset.dataContractId')

``` The field priority_reason will be used by TrackMe to display an informational message ```
| eval priority_external='datasets.splunkDataset.dataObservability.criticity.priority', priority_reason="data-contacts-id: " . 'datasets.splunkDataset.dataContractId'

``` Finally, the KVstore records will be updated; schedule this search so that any newly discovered entity will retrieve the expected externally managed priority ```
``` make sure to remove the complex fields coming from the YAML data-contracts ```
| fields - datasets.*, yaml-data
| outputlookup append=t key_field=keyid trackme_dsm_tenant_secops

TrackMe will automatically detect and apply the external priority management:

screen007.png

A similar technique could be used to manage further information, such as the max delay or latency thresholds. (data_max_delay_allowed, data_max_lag_allowed)