Remote Splunk deployments¶
A tenant can monitor a remote Splunk deployment — Enterprise or Cloud — instead of (or as well as) the local search head. One TrackMe instance then becomes a single pane of glass across many Splunk estates. Remote access is configured once as a remote account and referenced by tenants and trackers.
Which components need a remote account?¶
It depends less on the component than on where its data is. A component reads one of two things, and each has its own rule:
Indexed data (
_internal,_introspection,_audit, metrics, your own indexes). If the search head TrackMe runs on can search those indexes, the component useslocaland needs no remote account. Otherwise, it uses a remote account to a search head that can.A Splunk instance’s own REST API (its saved searches, users and roles, inputs, HEC tokens). The instance TrackMe runs on is
local. Any other search head tier or instance needs a remote account, even when it searches the same indexers.
What each component’s account points at¶
Component |
Remote account |
It points at |
|---|---|---|
Only when the data is on another deployment. |
A search head of that deployment. |
|
Optional, chosen per tracker. |
A search head that can search the tracker’s data. |
|
Rarely. VOL reads |
A search head of another deployment. It does not need to reach the license manager. |
|
One per search head tier other than TrackMe’s own. |
That tier: the saved searches’ definitions and versions are read there, even over the same indexers. |
|
One per search head tier other than the local one; remote accounts only for central monitoring. |
Each tier. The indexing-layer searches go through the route account, |
|
Only when the endpoints’ internal indexes are on another deployment. Optional accounts for the splunkd API enrichment. |
That deployment’s search tier, and the endpoints’ management port for the enrichment. |
|
Only when the collectors’ internal indexes are on another deployment. Optional, per collector, for the input inventory. |
That deployment’s search tier, and each collector for the inventory. |
|
Hybrid trackers, lookups, coverage gaps, CMDB enrichment |
Optional. |
The deployment that holds the data, lookup or CMDB. |
Always. |
Another TrackMe instance. |
|
No: Cribl tenants use their own Cribl connection. |
The Cribl Leader. |
Common setups¶
One search head, one indexing layer. No remote account. Every component reads locally; only the optional API reads of the Forwarder Collectors and Splunk HEC take an account per instance.
TrackMe next to Enterprise Security. TrackMe on the ad hoc search head, Enterprise Security on another tier over the same indexers: one remote account, to the ES tier, which WLK and UAM use to read its saved searches, users and roles. DSM, DHM, MHM, FLX, FQM and VOL stay local.
A dedicated monitoring Splunk. TrackMe on its own Splunk, watching other deployments: one remote account per deployment, used by that deployment’s tenants. VOL, UAM, Splunk HEC and Forwarder Collectors take one tenant per deployment.
Splunk Cloud, forwarders on your network. Your data is indexed on the stack, so the data components stay local. A premium search head (Enterprise Security) takes one account for WLK and UAM. Instances on your own network, read through their API, are reached through a TrackMe Broker, which only connects out.
Tip
Trackers run as the tenant owner, so the owner must hold one of the account’s allowed
roles (see Access control). The account itself needs read access to the indexes the
component searches, _internal included.
Remote accounts¶
A remote account stores everything TrackMe needs to reach a remote search head:
a name — the identifier tenants and trackers reference;
one or more Splunk REST URLs;
a bearer token (HTTPS only, stored encrypted in Splunk’s credential store);
the remote app namespace (defaults to
search);the allowed Splunk roles (account-level RBAC).
The implicit account local represents on-box data.
High availability and resilience¶
Failover — the URL field accepts a comma-separated list of REST endpoints. TrackMe tests them and picks a reachable one at random, so a search-head cluster with no load balancer still gets transparent failover.
Retry with backoff — transient connection and
5xxerrors are retried with exponential backoff (configurable attempts, initial delay, and multiplier per account).Connectivity test — building or editing an account runs a connectivity and authentication check, and refuses to save on failure.
All remote searches flow through the splunkremotesearch command, which TrackMe uses
transparently: it splits an SPL search into a remote part (run on the remote SH) and a
local part, so trackers work against remote data without any special handling.
Automatic token rotation¶
TrackMe rotates each account’s bearer token automatically — by default every 7 days.
Once a day, for each account, it creates a new token on the remote, verifies it, stores it
locally, and revokes the old one. The remote service account therefore needs
edit_tokens_own or edit_tokens_all, in addition to read access to its indexes (including
_internal). list_tokens_own is not needed to rotate: TrackMe lists the account’s tokens once
the rotation is done. Without that capability the new token is in place and the old one revoked, but this
last step fails, so the run reads as failed, including a rotation you force by hand.
Note
The initial token you create by hand is never revoked by TrackMe — disable it yourself once the first rotation has succeeded, per your token policy. A failed rotation is non-destructive: it is logged and the existing token stays in use.
You can also force a rotation on demand — for all accounts or a single one — from the Remote Accounts Overview. And for Splunk Cloud accounts, TrackMe can auto-heal the target URL list as the search-head-cluster members are recreated. Both are covered in Remote deployments — in depth.
Access control¶
Each account lists the local Splunk roles permitted to use it (rbac_roles) — by both
membership and inheritance. A user not in a listed role is refused by
splunkremotesearch. This prevents a tenant from being pointed at an unrelated Splunk to
exfiltrate data. See Roles & access control.
A tenant selects its remote account at creation time (and trackers and Flex objects can each target a remote account too); the same connectivity check runs automatically.
A deployment the stack cannot reach¶
From Splunk Cloud, a Heavy Forwarder, a Deployment Server or a search head on your own network
cannot be reached directly. Set Connectivity on the account to a TrackMe Broker: an add-on on a forwarder on your side then carries the account’s calls, and
every connection starts on your side. Nothing else about the account changes, and
splunkremotesearch works the same way.
The account’s token rotates automatically through the broker, as on a direct account and on the same
condition: the account on the destination can edit its tokens (edit_tokens_own or
edit_tokens_all). It is not the forwarder’s own token: two independent tokens are at work, and rotating one never touches the other.
See also
Remote deployments — in depth — cross-app RBAC, the REST endpoints, token-rotation internals, troubleshooting, and an SPL cookbook.
TrackMe Broker — reaching a deployment on your own network from Splunk Cloud.
Creating a tenant — choosing a deployment target when creating a tenant.
Roles & access control — account-level access control.
Lookup monitoring, Hybrid trackers and Replica trackers — features that also run against remotes.