Lookups monitoring — in depth

Tip

This is the in-depth reference for monitoring Splunk lookups (CSV and KV Store) as DSM entities. For the readable overview, start with Lookups monitoring. This page covers the dedicated TA, the KV Store mtime REST probe, the emitted entity fields, and the limitations and edge cases.

Introduction

Lookups monitoring is a dedicated search mode of the Hybrid Trackers wizard that turns Splunk lookups — CSV files and KVstore collections — into first-class TrackMe DSM entities.

Lookups are everywhere in a mature Splunk deployment: threat-intel feeds, asset databases, enrichment tables, pre-calculated aggregations, exclusion lists. When a lookup silently stops being updated, anything that relies on it degrades quietly — detections miss new IOCs, dashboards show stale enrichment, downstream alerts misfire. Detecting that early is exactly the same problem TrackMe solves for indexed data, and the lookups search mode applies the same DSM machinery to lookup objects.

What the tracker does on every run:

• Enumerates the lookups visible to the dispatching user (CSV, .csv.gz, KVstore collections) and filters them by app namespace and a name regex.

• For each CSV file: reads the filesystem modification time (os.stat) and streams the file to count rows — not the unreliable updated mtime returned by /services/data/lookup-table-files (which reflects the knowledge-object mtime and changes whenever permissions or props are touched, not when the file actually changed).

• For each KVstore collection: reads the document count (cursor-paginated, uncapped by default) and probes an operator-supplied, preference-ordered list of candidate timestamp fields to derive the modification time.

• Emits one DSM entity per lookup, with data_index = "lookups", data_sourcetype = "<app>:<name>" and object = "lookups:<app>:<name>".

• Alerts when a lookup has not been updated within the configured delay threshold (data_max_delay_allowed) — for example 86400 seconds for “must be updated at least once every 24 hours”.

Note

The previous approach using an Elastic Source with | from lookup:<name> | eval _time=... (see Elastic sources, Elastic source example 2) is superseded by this search mode. The lookups search mode is significantly faster on large lookups, does not require the lookup itself to maintain a _time field, and reads the actual filesystem mtime for CSV files rather than a per-row event time.

Why a dedicated Splunk add-on?

Lookups monitoring is powered by the trackmelookupsmonitor generative custom search command, shipped as a small dedicated Splunk Technology Add-on: TA-trackme-lookupmonitor.

The TA is split out from the main TrackMe app for three reasons:

1. AppInspect / Splunk Cloud certification — filesystem read access stays narrowly declared in a single-purpose TA. TrackMe itself does not gain any new sensitive permissions.

2. Lightweight remote search-head footprint — when monitoring lookups that live on a remote SH (separately-managed cluster, governance restrictions, etc.), only this small TA needs to be installed there. TrackMe itself is not required on the remote SH.

3. Standalone usability — the | trackmelookupsmonitor command can be invoked directly for ad-hoc lookup inventory, debugging or one-off audits.

Prerequisites

Install the TA-trackme-lookupmonitor add-on on every search head that will run a lookups tracker:

• The local search head where TrackMe is installed (mandatory — the tracker dispatches from there).

• Every remote search head referenced by a Splunk Remote Deployment account that you want to monitor lookups on (see Remote Splunk deployments). The command is wrapped by splunkremotesearch so it executes on the remote SH; the TA is what makes that wrapped call resolvable.

Download the TA from the TrackMe distribution site:

• https://downloads.trackme-solutions.com/TA-trackme-lookupmonitor/

The TA is built with the Splunk UCC framework and bundles splunklib — no Splunk SDK install is required. It exposes a single configuration page with logging-level controls; logs land in $SPLUNK_HOME/var/log/splunk/trackmelookupsmonitor.log.

Tip

The TA can be tested in isolation. Once installed, run a standalone search like this on the target search head — no TrackMe required:

| trackmelookupsmonitor app_namespace="search" lookup_type="csv"

Creating a lookups Hybrid Tracker — CSV example

The lookups search mode is selected from the standard Hybrid Trackers wizard. The first few steps (tracker name, target Splunk deployment) are identical to the other search modes — see Hybrid trackers for the wizard entry points and shared fields.

On Step 2 (Search mode and constraints), choose lookups (CSV file & KVstore monitoring) from the dropdown. A banner reminds you that the mode requires the TA-trackme-lookupmonitor add-on on the dispatching search head (and on the remote search head when a remote account is selected).

The wizard then hides the SPL constraint editor and replaces it with the lookups filter inputs:

• Lookup type — chosen first: csv (file based lookups), kvstore (collections) or both. It comes first because the Populate available lookups picker below only lists lookups of the selected type.

• App namespace(s) — a single app name, a glob (e.g. TA-*), or a comma-separated list mixing both (e.g. search, TA-*, Splunk_*). The dash - or asterisk * means “every app the dispatching user can see”. Whitespace around list items is ignored.

• Lookup name pattern — a regular expression applied (re.search) to the lookup name. Backslash escapes such as \d, \w, \. are preserved.

Dynamic pickers (2.4.8)

You no longer have to hand-write the namespace globs or the name regex. Two buttons discover what is actually available on the target deployment — local or remote — and translate your selection back into the fields above:

• Populate available app namespaces — lists the app namespaces present on the deployment; pick them from a searchable list and the selection is written into App namespace(s), so there is no need to type globs by hand.

• Populate available lookups — lists the CSV files and/or KVstore collections of the selected lookup type; pick exactly the ones to monitor and TrackMe generates the matching Lookup name pattern for you — no manual pattern authoring.

Once you pick from the two lists, the chosen apps appear as chips and are written into App namespace(s) as a comma-separated list, and the selected lookups fill Lookup name pattern with an anchored regex matching exactly those lookups (the picker also narrows the app namespaces to just the apps of your selection). The list shows a capped preview — e.g. 20 of 297 shown — but the selection itself is exact:

Steps 3 (Break by logic) and 4 (Time ranges) are repurposed in lookups mode:

• Break by logic is disabled with an explanatory banner — there is no break-by in lookups mode, the tracker creates one entity per discovered lookup.

• Time ranges are replaced with a single numeric field, “Alert if the lookup has not been updated in the past (seconds)”. The default is 86400 (24 hours). This value is stored on the entity as data_max_delay_allowed and is the primary alerting knob. Latency-based alerting does not apply to lookups (the tracker reads the actual filesystem mtime, not an ingestion timestamp) so the Splunk earliest/latest pickers are deliberately hidden.

Creating a lookups Hybrid Tracker — KVstore example

The KVstore flow is identical to the CSV one — including the two Populate pickers — with one extra input: KVstore time field(s). Set Lookup type to kvstore (KVstore collections) and the Populate available lookups picker lists collections instead of files.

Why does KVstore need an extra input? splunkd only maintains _key and _user automatically on KVstore documents. There is no automatic _time field — it is only present if the storing application explicitly sets it. Conventions vary widely (_time, mtime, updated_at, modified, timestamp, last_modified…), so the tracker needs to know which field(s) carry the per-record modification time of your collections.

The default candidate list — _time, mtime, updated_at, modified, timestamp, last_modified — covers the most common conventions. The probe is preference-ordered: the tracker tries each candidate in order, runs

GET /servicesNS/-/<app>/storage/collections/data/<collection>
    ?fields=<field>&sort=<field>:-1&limit=1

and stops at the first candidate that returns a parseable value (epoch number, numeric string, or ISO 8601 timestamp). The chosen field is recorded as mtime_source = "kvstore:<field>" on the emitted entity, so operators can confirm which field was read.

When none of the candidates work — i.e. the collection has no time-bearing field that the tracker can parse — mtime_source is set to "unavailable" and data_last_time_seen is null. The entity is still created (so its volume can be tracked) but the delay-state evaluation cannot trigger.

Step 5 (Test and Review) — simulating the discovery

The simulation step runs the discovery pipeline in preview mode and renders a results table with up to 100 entities so you can verify the filter scope and the resolved attributes before committing. The total in the success notification reflects the true number of matched lookups, even when more than 100 are present (the table itself is capped to keep the wizard responsive).

CSV example — a single lookup matching the filter:

What to look at on the simulation row:

• object — the entity identifier (lookups:<app>:<name>). This is what will appear in the DSM entity inventory.

• app_namespace — the Splunk app the lookup lives in.

• lookup_type — csv, csv_gz, kvstore, or other for non-CSV file payloads (.kmz, .geo).

• lookup_path — the absolute filesystem path for CSV; <app>:<collection> for KVstore.

• data_eventcount — the record count (rows for CSV, documents for KVstore).

• lookup_size_bytes — the on-disk size of the CSV file (os.stat size); not applicable to KVstore collections.

• data_last_time_seen — the resolved modification time (epoch seconds).

• mtime_source — fs for CSV (filesystem os.stat mtime), kvstore:<field> for KVstore (the candidate field that resolved), or unavailable.

KVstore example — single collection, mtime resolved from a candidate field:

In the KVstore example, mtime_source = "kvstore:last_time" — the tracker probed the candidate list and last_time was the first candidate to yield a parseable value.

Tip

If your simulation returns zero rows, the most common causes are an over-restrictive app_namespace glob, a name_pattern that does not match any visible lookup, or — for KVstore — a candidate list that does not include the field your collections actually use. The simulation SPL is printed in the wizard so you can run it directly in the search bar and adjust.

The resulting DSM entities

Each discovered lookup becomes a standard DSM entity, with data_index = "lookups" (a pseudo-index that classifies these entities in the inventory) and data_sourcetype = "<app>:<name>" (unique per lookup).

Because lookups have no _indextime concept, the ingestion latency sub-system of DSM is neutralised on lookup entities — there is no meaningful latency to measure between “the file was modified” and “Splunk indexed it”. Lookup entities therefore get a normalised, lookup-native presentation (2.4.8): the Overview leads with a LAST KNOWN RECORDS IN LOOKUP count tile, the latency tiles are hidden, and the data-source chart plots number of records / update frequency rather than volume/latency. The same normalisation carries through the status messages, the entity charts and the stateful-alert emails, which all speak records and update frequency instead of feed metrics.

The primary alerting signal is the delay since the last modification, gated by the data_max_delay_allowed threshold configured at tracker creation time (or edited later from the entity’s Update frequency policy panel).

CSV-based lookup entity — note the “LAST KNOWN RECORDS IN LOOKUP” tile:

KVstore-based lookup entity:

Both entity views expose the standard DSM tabs: Overview, Incidents, Performance Metrics, Outliers Anomaly Detection, Data Sampling, Status Flipping, Status Message, SLA, Audit Changes, Handlers Activity. The Outliers detection on data_eventcount works out of the box — if a lookup that normally has ~50,000 rows drops to 200 (or balloons to 5,000,000) the standard outliers engine flags it without any extra configuration.

Configuring staleness alerting — the Update frequency policy

The staleness threshold and its impact-score weight are edited from the entity’s Actions menu → Lagging policies:

For lookup entities the screen is normalised to lookup semantics and titled Update frequency policy (lookups) — the latency inputs that only apply to indexed feeds are removed, and the delay controls are reframed as update frequency / staleness:

• Maximum allowed time since last update — the staleness threshold (the same value as data_max_delay_allowed). When a variable-delay policy is active it is managed by time-based slots; use Configure variable delay to edit those.

• Staleness impact score weight — a custom impact-score weight (0–100) applied when the lookup exceeds the allowed time since last update. This is the direct equivalent of the delay weight on feed entities; if left unset the tenant-level default is used (here it inherits 100).

• Threshold lock — Unlocked lets TrackMe auto-manage the threshold (adaptive delay / lagging classes); Locked means you manage it manually and TrackMe never auto-adjusts it.

• Future tolerance and the Advanced tools (thresholds simulation, auto-definition helpers, configure variable delay) behave exactly as they do for feed entities.

When a lookup is refreshed within its allowed window the entity stays green: the Status message confirms the lookup was updated within the maximum allowed staleness threshold and reports the current record count, and the impact score stays at 0 with an empty score-components list:

When the lookup goes past its allowed window, the entity flips red and the staleness impact-score weight is applied. The Status message states how long it has been since the last update and by how much it exceeds the threshold. Here the anomaly reason is variable_delay_threshold_breached because a variable-delay policy with time-based slots is active; with a fixed threshold it reports delay_threshold_breached:

Minimum records — alerting on empty or under-filled lookups

(2.4.8) Beyond staleness, TrackMe can alert when a lookup runs empty, or drops below a minimum record count — catching the case where a lookup is still updated on schedule but has silently lost its content (a rebuild that produced no rows, a truncated export, a broken upstream job).

Note

This is on by default: lookup entities require at least one record. When the minimum is left unset it defaults to 1, so an empty lookup (0 records) flips red out of the box — no configuration needed. The record count (rows for CSV, documents for KVstore) is compared against the threshold on every run; when it is lower, the entity turns red with the anomaly reason min_records_not_reached.

Set the threshold per entity from the Actions menu → Minimum records (lookups):

• A positive integer N requires at least N records.

• ``0`` disables the check entirely (for lookups that are legitimately allowed to be empty).

• Unset falls back to the default of 1 — i.e. alert on zero records.

When the condition is not met, the entity flips red, min_records_not_reached is added to anomaly_reason, and a min_records_breach component is added to the impact score (weight 100 by default, tunable per tenant via impact_score_dsm_min_records_breach):

The minimum-records and staleness checks are independent — the Status message spells out both, and the impact-score breakdown shows the min_records_breach component:

How modification times are resolved

CSV lookups

The tracker calls /services/data/lookup-table-files to enumerate visible lookups and resolve each one’s on-disk path (content.eai:data), then calls os.stat(path) to read the filesystem mtime. That is deliberately not the updated mtime that the REST endpoint exposes — that one reflects the knowledge-object mtime (touched whenever permissions, ACLs, or transforms.conf properties change) and is not a reliable proxy for “when did the file last change?”.

mtime_source = "fs" on the emitted entity signals this resolution path.

For .csv.gz files the same path is used; the row counter handles the gzip wrapper natively.

KVstore collections

For each KVstore collection that matches the filter, the tracker takes the comma-separated, preference-ordered KVstore time field(s) list and probes each candidate in order. The first candidate that returns a parseable value (epoch number, numeric string, or ISO 8601 — with Z, +HH:MM, +HHMM suffixes; naive timestamps treated as UTC) wins.

The emitted entity carries mtime_source = "kvstore:<field>" so the resolution is fully traceable. When no candidate works, mtime_source = "unavailable" and data_last_time_seen is null.

Document counts are obtained by cursor-walking the collection with fields=_key and summing the page lengths. By default the walk is uncapped — the kvstore_count_cap option defaults to 0 (no limit), so the full collection is counted. An operator can set a positive cap to bound the count on pathologically large collections; only then, when the cap is reached, is kvstore_count_truncated = 1 set on the emitted row. The cursor page size is tunable via kvstore_page_size (default 10000).

Remote search heads

The | trackmelookupsmonitor command is forced-local (local = true in commands.conf) so it always executes on the search head that dispatches it. To monitor lookups that live on a remote search head:

1. Install TA-trackme-lookupmonitor on the remote SH (TrackMe itself is not required there — that is the entire point of the TA split).

2. Configure a Splunk Remote Deployment account for that SH in TrackMe (see Remote Splunk deployments).

3. Select that account in Step 1 of the wizard. TrackMe will automatically wrap the lookups command with splunkremotesearch so the discovery runs on the remote SH and the resulting rows are streamed back for persistence.

The wrapping is transparent — you do not need to write SPL yourself. The wizard’s Test and Review step renders the final SPL (with the splunkremotesearch wrapper applied when relevant) so you can review it before saving.

Notes and limitations

• Permissions — the saved search runs as the tenant owner (the user configured on the Virtual Tenant). Lookups that are not visible to that user under the configured app namespace will not be discovered.

• Definition-only lookups — a lookup defined in transforms.conf with no on-disk CSV file yet (content.eai:data = None) is skipped with a warn-level audit message. This is rare but not impossible (test fixtures, broken installs).

• KVstore mtime requires an operator-maintained field — splunkd does not auto-maintain a timestamp on KVstore documents. If your collections have no time-bearing field, the entity will still be created (so its volume can be tracked via outliers / threshold rules on data_eventcount) but the delay-state evaluation cannot fire — mtime_source will be unavailable. This is documented in the tracker output so it is never silent.

• Large CSV row count cost — the row counter is an O(filesize) streaming scan. A 10 GB lookup on local NVMe completes in 10-20 seconds; this is acceptable on minute-scale schedules but worth keeping in mind on multi-GB .csv.gz lookups (decompression cost dominates). Lookups are inspected in parallel to keep discovery fast over many objects — the concurrency is controlled by max_workers (default min(cpu*2, 8)), which you can lower to cap the load on a busy search head.

• Stat / count race — modification time is read first, row count second. Under a concurrent write, the worst-case result is “new mtime + previous-cycle count” for a single tick. This is acceptable and self-corrects on the next run.

See also

• Hybrid trackers — Hybrid Trackers wizard entry points and shared fields.

• Remote Splunk deployments — configuring remote Splunk accounts (required for remote-SH lookups monitoring).

• Elastic sources — the Elastic source example 2: tracking lookups update and number of records section is now superseded by the lookups search mode documented here.

• TA-trackme-lookupmonitor download page — dedicated add-on distribution site.