Remote Splunk deployments

Two ways to reach a remote Splunk deployment Two scenarios. First, direct: TrackMe reaches the remote Splunk deployment itself, over HTTPS on the Splunk REST API, with the account's bearer token. Second, a Splunk Cloud customer whose deployment is on their own network: the firewall lets nothing in, so a TrackMe Broker pool of two Heavy Forwarders on that network connects out to the stack over HTTPS on port 8089, picks up the calls, makes them against the remote Splunk deployment and returns the answers. The account only says how it connects, and its searches work the same way. Direct: TrackMe can reach the deployment TrackMe with a remote account HTTPS, Splunk REST API with the account's bearer token Remote Splunk Enterprise or Cloud Splunk Cloud customer: a deployment on your own network, through a TrackMe Broker Splunk Cloud TrackMe with a remote account Firewall Your network Connects out HTTPS, 8089 Picks up calls, returns answers Nothing connects in TrackMe Broker pool Heavy Forwarder TrackMe Broker add-on Heavy Forwarder TrackMe Broker add-on If one stops, the other carries on Target Remote Splunk Heavy Forwarder, Deployment Server, search head A remote account simply says how it connects: Direct, or through a broker. Nothing else about the account changes, and its searches work the same way.

A tenant can monitor a remote Splunk deployment — Enterprise or Cloud — instead of (or as well as) the local search head. One TrackMe instance then becomes a single pane of glass across many Splunk estates. Remote access is configured once as a remote account and referenced by tenants and trackers.

Which components need a remote account?

It depends less on the component than on where its data is. A component reads one of two things, and each has its own rule:

  • Indexed data (_internal, _introspection, _audit, metrics, your own indexes). If the search head TrackMe runs on can search those indexes, the component uses local and needs no remote account. Otherwise, it uses a remote account to a search head that can.

  • A Splunk instance’s own REST API (its saved searches, users and roles, inputs, HEC tokens). The instance TrackMe runs on is local. Any other search head tier or instance needs a remote account, even when it searches the same indexers.

Do you need a remote account? A decision in two branches. If the component reads indexed data, such as _internal, _introspection, _audit, metrics or your own indexes, ask whether the search head TrackMe runs on can search those indexes: if yes, use local and no remote account is needed; if not, use a remote account to a search head that can search them. If the component reads a Splunk instance's own REST API, such as saved searches, users and roles, inputs or HEC tokens, ask whether that instance is the search head TrackMe runs on: if yes, use local; if not, use one remote account per search head tier or instance. When another search head shares the same indexers, the data stays local and only the REST reads need an account. What does the component read? Indexed data _internal, _introspection, _audit, metrics, your indexes A Splunk instance's own REST API saved searches, users and roles, inputs, HEC tokens Can the search head TrackMe runs on search those indexes? Is that instance the search head TrackMe runs on? Yes local no remote account No A remote account to a search head that can search them Yes local no remote account No A remote account per search head tier or instance Same indexers, another search head? The data stays local: only the REST reads need an account.

What each component’s account points at

Which components use a remote account, and what it points at Every TrackMe component and feature, grouped by what its remote account points at. Only if the data is elsewhere, a remote account to a search head that can search the data: DSM, DHM, MHM, FLX, FQM, VOL, Splunk HEC, Forwarder Collectors, hybrid trackers, lookups and coverage gaps, CMDB enrichment. One per search head tier other than TrackMe's own, even over the same indexers: WLK and UAM. Optional, one per instance, to read its configuration through its own API: the Forwarder Collectors input inventory and the Splunk HEC splunkd API enrichment. Always, a remote account to the other TrackMe: replica trackers. Not a remote account: Cribl Stream and Cribl Edge use their own Cribl connection. From Splunk Cloud, an instance on your own network is reached through a TrackMe Broker. Only if the data is elsewhere A remote account to a search head that can search the data DSM DHM MHM FLX FQM VOL Splunk HEC Forwarder Collectors Hybrid trackers Lookups and coverage gaps CMDB enrichment One per search head tier Every tier other than TrackMe's own, even over the same indexers WLK UAM Optional, one per instance Reads an instance's configuration through its own API Forwarder Collectors: input inventory Splunk HEC: splunkd API enrichment Always A remote account to the other TrackMe Replica trackers Not a remote account Cribl tenants use their own Cribl connection Cribl Stream Cribl Edge From Splunk Cloud, an instance on your own network is reached through a TrackMe Broker.

Component

Remote account

It points at

DSM, DHM, MHM

Only when the data is on another deployment.

A search head of that deployment.

FLX, FQM

Optional, chosen per tracker.

A search head that can search the tracker’s data.

VOL

Rarely. VOL reads license_usage.log in _internal on the indexers the license manager forwards to, so local works wherever TrackMe searches those indexers.

A search head of another deployment. It does not need to reach the license manager.

WLK

One per search head tier other than TrackMe’s own.

That tier: the saved searches’ definitions and versions are read there, even over the same indexers.

UAM

One per search head tier other than the local one; remote accounts only for central monitoring.

Each tier. The indexing-layer searches go through the route account, local when it is selected.

Splunk HEC

Only when the endpoints’ internal indexes are on another deployment. Optional accounts for the splunkd API enrichment.

That deployment’s search tier, and the endpoints’ management port for the enrichment.

Forwarder Collectors

Only when the collectors’ internal indexes are on another deployment. Optional, per collector, for the input inventory.

That deployment’s search tier, and each collector for the inventory.

Hybrid trackers, lookups, coverage gaps, CMDB enrichment

Optional.

The deployment that holds the data, lookup or CMDB.

Replica trackers

Always.

Another TrackMe instance.

Cribl Stream and Cribl Edge

No: Cribl tenants use their own Cribl connection.

The Cribl Leader.

Common setups

Remote accounts in four common setups Four common setups. One search head over one indexing layer: no remote account, everything reads locally; optionally one account per instance for the Forwarder Collectors and Splunk HEC API reads. TrackMe on the ad hoc search head next to an Enterprise Security search head over shared indexers: one remote account, to the ES tier, for WLK and UAM; the data components stay local. A dedicated monitoring Splunk: one remote account per monitored deployment, used by its tenants, and one VOL, UAM, Splunk HEC or Forwarder Collectors tenant per deployment. Splunk Cloud with forwarders on your own network: the data is on the stack so data components stay local, the premium search head takes one account for WLK and UAM, and instances on your network are reached through a TrackMe Broker, which connects out. One search head No remote account Search head TrackMe runs here Indexers Everything reads locally. Optional: one account per instance for the Forwarder Collectors and Splunk HEC API reads. Next to Enterprise Security 1 remote account Ad hoc search head TrackMe runs here ES search head another tier remote account Shared indexers One account, to the ES tier: WLK and UAM read its saved searches, users and roles. DSM, DHM, MHM, FLX, FQM and VOL stay local. Dedicated monitoring Splunk 1 per deployment TrackMe its own Splunk Deployment A search head Deployment B search head remote accounts One account per deployment, used by its tenants. VOL, UAM, Splunk HEC and Forwarder Collectors: one tenant per deployment. Splunk Cloud + your network 1 account + Broker Splunk Cloud Ad hoc SH TrackMe Premium SH ES Indexers Your network Forwarders + TrackMe Broker connects out nothing in Your data is on the stack: data components stay local. Premium search head: one account (WLK, UAM). Instances on your network: accounts through a Broker.

One search head, one indexing layer. No remote account. Every component reads locally; only the optional API reads of the Forwarder Collectors and Splunk HEC take an account per instance.

TrackMe next to Enterprise Security. TrackMe on the ad hoc search head, Enterprise Security on another tier over the same indexers: one remote account, to the ES tier, which WLK and UAM use to read its saved searches, users and roles. DSM, DHM, MHM, FLX, FQM and VOL stay local.

A dedicated monitoring Splunk. TrackMe on its own Splunk, watching other deployments: one remote account per deployment, used by that deployment’s tenants. VOL, UAM, Splunk HEC and Forwarder Collectors take one tenant per deployment.

Splunk Cloud, forwarders on your network. Your data is indexed on the stack, so the data components stay local. A premium search head (Enterprise Security) takes one account for WLK and UAM. Instances on your own network, read through their API, are reached through a TrackMe Broker, which only connects out.

Tip

Trackers run as the tenant owner, so the owner must hold one of the account’s allowed roles (see Access control). The account itself needs read access to the indexes the component searches, _internal included.

Remote accounts

A remote account stores everything TrackMe needs to reach a remote search head:

  • a name — the identifier tenants and trackers reference;

  • one or more Splunk REST URLs;

  • a bearer token (HTTPS only, stored encrypted in Splunk’s credential store);

  • the remote app namespace (defaults to search);

  • the allowed Splunk roles (account-level RBAC).

The implicit account local represents on-box data.

High availability and resilience

  • Failover — the URL field accepts a comma-separated list of REST endpoints. TrackMe tests them and picks a reachable one at random, so a search-head cluster with no load balancer still gets transparent failover.

  • Retry with backoff — transient connection and 5xx errors are retried with exponential backoff (configurable attempts, initial delay, and multiplier per account).

  • Connectivity test — building or editing an account runs a connectivity and authentication check, and refuses to save on failure.

All remote searches flow through the splunkremotesearch command, which TrackMe uses transparently: it splits an SPL search into a remote part (run on the remote SH) and a local part, so trackers work against remote data without any special handling.

Automatic token rotation

TrackMe rotates each account’s bearer token automatically — by default every 7 days. Once a day, for each account, it creates a new token on the remote, verifies it, stores it locally, and revokes the old one. The remote service account therefore needs edit_tokens_own or edit_tokens_all, in addition to read access to its indexes (including _internal). list_tokens_own is not needed to rotate: TrackMe lists the account’s tokens once the rotation is done. Without that capability the new token is in place and the old one revoked, but this last step fails, so the run reads as failed, including a rotation you force by hand.

Note

The initial token you create by hand is never revoked by TrackMe — disable it yourself once the first rotation has succeeded, per your token policy. A failed rotation is non-destructive: it is logged and the existing token stays in use.

You can also force a rotation on demand — for all accounts or a single one — from the Remote Accounts Overview. And for Splunk Cloud accounts, TrackMe can auto-heal the target URL list as the search-head-cluster members are recreated. Both are covered in Remote deployments — in depth.

Access control

Each account lists the local Splunk roles permitted to use it (rbac_roles) — by both membership and inheritance. A user not in a listed role is refused by splunkremotesearch. This prevents a tenant from being pointed at an unrelated Splunk to exfiltrate data. See Roles & access control.

A tenant selects its remote account at creation time (and trackers and Flex objects can each target a remote account too); the same connectivity check runs automatically.

A deployment the stack cannot reach

From Splunk Cloud, a Heavy Forwarder, a Deployment Server or a search head on your own network cannot be reached directly. Set Connectivity on the account to a TrackMe Broker: an add-on on a forwarder on your side then carries the account’s calls, and every connection starts on your side. Nothing else about the account changes, and splunkremotesearch works the same way.

The account’s token rotates automatically through the broker, as on a direct account and on the same condition: the account on the destination can edit its tokens (edit_tokens_own or edit_tokens_all). It is not the forwarder’s own token: two independent tokens are at work, and rotating one never touches the other.

See also