TrackMe Broker — security and cryptography¶

Tip

This page is for whoever needs to assess the security of TrackMe Broker: how a forwarder and a stack come to trust each other, every key and cipher, and what holds against what. For the concept and the setup, start with TrackMe Broker. For how a call travels, see TrackMe Broker — in depth.

The short version¶

The customer’s side starts every connection. Nothing listens on the forwarder for the stack, and no inbound rule is needed.
A Splunk token alone cannot pull or push: it can ask to be accepted, and only an administrator entering a short-lived code accepts it.
From then on every call the forwarder makes is signed, and every request it receives is signed by the stack for that very pull.
The forwarder’s Splunk token renews itself, at the forwarder’s signed request, so a stolen token alone cannot do it: see the forwarder’s Splunk token.
Requests and responses are encrypted between the stack and the forwarder, for the other side’s key, on top of TLS.
In the KV store, requests and responses are encrypted with the broker’s own key.
The forwarder reaches only the targets its administrator declared. The stack cannot widen that list, and never receives it.
Every primitive is FIPS-approved, on one curve. On a forwarder they run under Splunk’s FIPS module. On the stack TrackMe uses the library it packages, rather than Splunk’s module: see FIPS.

What it does	Primitive	Sizes
Signs channel calls, hand-overs, refusals, and answers to a rotation or a token renewal	ECDSA on P-256 with SHA-256, DER-encoded, then base64	256-bit key
Agrees a key for one seal	Ephemeral-static ECDH on P-256	32-byte shared secret, 65-byte uncompressed points
Derives the key of one seal	HKDF-SHA256	No salt, 32-byte output
Encrypts between the two sides	AES-256-GCM	32-byte key, 12-byte random nonce, 16-byte tag
Binds keys to the pairing code	HMAC-SHA256	32-byte key derived from the code, 64 hex characters of output
Encrypts rows at rest on the stack	Fernet: AES-128-CBC with HMAC-SHA256	32-byte key: 16 to encrypt, 16 to authenticate
Fingerprint, digests	SHA-256	64 bits shown for a fingerprint

The library is cryptography. On the forwarder it comes from Splunk’s own Python: the add-on does not package it. On the stack, TrackMe packages its own, with its own OpenSSL. The stack and the add-on hold the same implementation of pairing, signing and sealing. Fernet is on the stack only.

The keys¶

Key	Held by	Purpose
Instance signing key, ECDSA P-256	The forwarder’s `storage/passwords`, realm `trackme_broker_pairing`, one per connection	Signs every channel call
Instance sealing key, ECDH P-256	The same	Opens what the stack seals for this instance
Stack signing key, ECDSA P-256	The stack’s `storage/passwords`, realm `trackme_broker`, one per deployment	Signs what it hands to an instance, its refusals, and its answers to a rotation or a token renewal
Stack sealing key, ECDH P-256	The same	Opens responses sealed for the stack
Broker key, Fernet	The stack’s `storage/passwords`, realm `trackme_broker`, one per broker	Seals rows at rest in the KV store

A key is never used for both signing and sealing. Private keys never leave their side. The stack creates its keys at the first acceptance, so a deployment with no broker holds none, and every search head reads the same ones.

Keys on a search head cluster¶

TrackMe Broker works on a search head cluster, on Splunk Cloud too: any member can serve a forwarder’s call. A cluster replicates credentials when it can, and nothing says how long that takes on a given deployment. So no step here waits for a time or assumes one. On the stack, a key is in two places, and each decides one thing.

The credential store holds the secret, under a name made for this one creation: <name>.<key_id>, with a random id. So search heads never overwrite each other.
A row of kv_trackme_broker_pairings, key:<name>, says which credential is the key. One insert wins, on any member, and it is written only after the credential exists. Whoever loses removes their own credential.

No row means no key, and the next write makes one. A row whose credential is not on this search head yet means the key is on its way. The caller hears key_replicating, a 503, and nothing else changes, for however long that lasts. Each caller adapts, with no limit of its own:

Who is asking	What it does
A queued request	It stays queued. The stack opens a row before it claims it, so a key on its way leaves no claim behind.
A call being queued	Nothing was queued. TrackMe asks again for as long as the call’s own time lasts, and then fails it as not sent.
A caller reading a result or a chunk	It treats this as passing, like any other failed poll.
A forwarder’s `pull` or `push`	It gets a 503 and backs off, and its page says why. A `push` or a `fail` is retried until it lands or its claim ends.
Add broker	It asks again by itself and says it is waiting, until the keys arrive, the code runs out or the administrator closes the dialog.

A credential that is truly gone, deleted by hand or lost with a member destroyed before it replicated, reads as key_replicating for good, because nothing can tell it from a slow replication. For a broker, Replace sealing key writes a new credential and points the row at it. For the stack’s keys, see keys that live longer.

Only a conflict proves that a row was lost, so only a conflict removes a credential. A KV write whose answer is lost may still have been written, so it removes nothing, and a Replace sealing key that fails removes neither credential. A crash or a lost answer can therefore leave a credential that nothing names. It seals nothing and stands in nobody’s way.

Replacing a broker’s sealing key can fail what is in flight for up to a minute, the time a process caches a key: a search head can go on sealing with the old key until its copy refreshes, and what it sealed then ends as sealed_unavailable.

Pairing and key exchange¶

splunkd authenticates every call to the stack’s REST API, so the Splunk token stays, and pairing does not replace it. Pairing adds that a token alone gets nothing.

The steps¶

Keys. Saving a connection on the add-on’s page creates the input. It generates two P-256 key pairs, one to sign and one to seal, and stores them in the forwarder’s credential store.
The code. It generates a pairing code: 20 characters of Crockford base32, so 100 bits from a cryptographic random source, grouped for reading as 7KQ4-MZ2P-X9TB-R3HV-6WNC. It is read case-insensitively, with O as 0, and I and L as 1. It lasts 15 minutes, and a new one replaces it.
The binding. The code never travels on the channel. What travels is binding = HMAC-SHA256(key, canonical registration), where the registration holds the add-on’s version, the host, the instance id, both public keys, a random nonce and the protocol. A row on the stack therefore holds a binding, not a code.
Register. The add-on sends the registration and the binding, with a signature by the new signing key over both, so a captured registration cannot be sent again under another code. The stack checks the signature and the sealing key’s point, and stores a pending row with the token’s account and the source address. It refuses a registration above 4 KB, and an account can hold at most 20 pending rows, so a token cannot fill the collection.
Carried by hand. The add-on’s page shows the code and the fingerprint. An administrator enters the code in Add broker on Manage TrackMe Brokers.
Preview. The stack looks for the pending row whose binding matches the code, comparing in constant time. So the keys it finds are the ones generated next to that code, and something else holding the same token cannot substitute its own. A code that more than one row claims is refused, pairing_code_ambiguous, rather than guessed: someone else holds it. The preview shows the fingerprint, the instance, the host, the source address, the Splunk account and the add-on’s version, and warns if this replaces a forwarder already paired under the same instance id. On Splunk Cloud, the source address is the load balancer’s and not your forwarder’s. Nothing has changed yet.
The fingerprint. The administrator compares it with the add-on’s page. It is the first 16 hex characters of the SHA-256 of both public keys, shown in four groups, on both sides. This is the check that a person makes between the two sides: it shows that what is about to be trusted is what the forwarder generated.
Accept. The acceptance names the row the preview showed, pairing_changed otherwise. One acceptance per row, on any search head: the first insert of a lock row wins, and a holder that died leaves it for two minutes. The stack pins the instance’s public keys and the Splunk account that registered them, and removes any earlier keys of that instance.
The answer. The stack builds answer = {broker, instance_id, stack: {signing, sealing}} and HMAC-SHA256(key, canonical answer), under the same code. It keeps the answer for seven days, so an acceptance is not lost to a code running out or a forwarder being away. The add-on always reads where its registration stands before registering again.
Pinning. The add-on polls pairing_status every 5 seconds, signed by the registration’s key. On accepted, it checks the binding under the code it generated, before pinning anything. So the stack’s keys are bound to the same code, whatever sits between the two, including a TLS-inspecting proxy. No match reads acceptance_not_bound_to_code, and nothing is pinned. It then forgets the code and starts pulling.

A second forwarder repeats the same steps, and joins the existing broker. Waiting to be accepted on Manage TrackMe Brokers lists pending registrations with no way to accept one from the list: only the code does.

How the code becomes a key¶

The HMAC key is SHA-256("trackme-broker-pairing-v1" || "\n" || code), with the code normalised first. The code holds 100 bits, and a FIPS mode can refuse an HMAC key under 112 bits, so it is hashed. Canonical means JSON with sorted keys, no spaces, and ASCII only, the same on both sides.

Why a token alone only registers¶

Every operation other than register and pairing_status goes through the same check, in this order:

The instance is paired, its broker exists, and its broker is enabled. Otherwise instance_not_paired or broker_disabled.
The token’s account is the one pinned at acceptance. Otherwise broker_user_refused.
ts is within 300 seconds of the stack’s clock. Otherwise clock_skew.
The pinned signing key verifies the call. Otherwise invalid_signature.

Only then does the handler act, on the system token, on the rows of the broker that the call names. register and pairing_status have their own checks.

The signed channel¶

What each signature covers¶

Each message starts with its own label, so a signature for one purpose can never serve for another. The last row is the exception: its message is the JSON of the new keys, and only the new key signs it.

Signature	Key	Message
A registration	Its own new signing key	`trackme-broker-registration-v1\n` + canonical `{binding, registration}`
`pairing_status`	Its own new signing key	`trackme-broker-status-v1\n<registration_id>`
A channel call	Instance signing key	`trackme-broker-call-v1`, operation, broker, instance id, `ts`, and the SHA-256 of the exact body string, joined by newlines
A request handed over	Stack signing key	`trackme-broker-request-v2`, request id, claim token, the pull’s nonce, attempt, remaining seconds, and the sealed value, joined by newlines
A refusal	Stack signing key	`trackme-broker-refusal-v1\n` + canonical `{broker, call, instance_id, operation, reason, server_time, ts}`
An answer to a rotation	Stack signing key	`trackme-broker-rotation-v1\n` + canonical `{broker, call, instance_id, keys, ts}`
An answer about a token	Stack signing key	`trackme-broker-token-v1\n` + canonical `{broker, call, detail, instance_id, token_id, ts}`. `detail` is the sealed token when handing one over, and then `confirmed`, `retired` or `kept`
The proof of new keys	The new signing key	Canonical `{instance_id, keys}`

In a refusal, and in an answer to a rotation or about a token, call is the SHA-256 of the signature of the call being answered, which nobody can know before the forwarder makes that call.

A request is bound to one pull¶

Every pull carries a fresh nonce: 16 random bytes, 32 hex characters. The stack signs each request it hands over for that nonce, with the seal, the claim, the attempt and the time left. The add-on refuses a request whose signature does not verify for the nonce it sent. So an answer captured on the way cannot be played back at a later pull, a day later, to run a POST again, and the budget cannot be changed on the way.

Signed refusals and clock correction¶

Two answers change what a forwarder holds: instance_not_paired makes it forget its keys, and clock_skew changes its clock offset. Anything on the way could otherwise do either with one forged answer. So each carries a proof: the stack’s signature over the reason, the operation, the broker, the instance, ts, the time it gives, and the digest of the call’s own signature.

The add-on acts only on a proof that verifies against the stack key it pinned, in answer to the call it just made. Anything else changes nothing: the keys stay, the page reads not_paired_unverified, and it tries again. A rebuilt stack holds other keys, so its refusals never verify, and the page says to remove the connection and add it again.

A clock that disagrees must not lock a broker out. On a proven clock_skew, HTTP 409, the add-on keeps the difference from the server_time in the answer, and sends the call once more. This is the same reason the add-on budgets a call from remaining_seconds. A hand-over also carries lease_until, an absolute time that is not signed, and the add-on ignores it.

Key rotation and revocation¶

An instance’s keys rotate every 30 days, without anyone acting. New keys are stored before the stack hears of them, so keys that could not be stored are never offered.
rotate_key is signed by the current key, and carries the new public keys with a proof: the new signing key’s signature over the instance and the keys. So the stack knows the sender holds both.
The stack keeps honouring the previous signing key for an hour, so a call already on its way is not refused. Only the current key can start a rotation: the previous one can only repeat the rotation that retired it, or a leaked old key would take the instance over.
A forwarder moves to new keys only on the stack’s signed answer, for this very call. One forged 200 would otherwise leave it signing with keys the stack has never seen, for good. A 200 with no valid proof reads rotation_not_confirmed, and it keeps the keys it has. A rotation that is refused, or that cannot reach the stack, keeps its own code, and is tried again 5 minutes later.
If the answer was lost, the add-on keeps the new keys as its next keys and repeats. A repeat changes nothing on the stack, and answers the same way. Once its old key is refused, more than an hour later, it signs the repeat with the new key, moves on the signed answer, and sends the call again.
The add-on tries its current, next and previous sealing keys when it opens a request, because one sealed just before the rotation may still arrive.

Keys that live longer¶

The stack’s own keys do not rotate, and nothing on Manage TrackMe Brokers replaces them. To replace them, because they leaked or because their credential is truly gone, delete the row key:__stack__ of kv_trackme_broker_pairings, then pair every forwarder again: forwarders pin those keys, so on each one remove the connection and add it again, which takes a token for each. The next acceptance makes new keys. After pairing again, each forwarder keeps its earlier token until its next renewal, and then holds one.

The broker’s Fernet key changes when an administrator selects Replace sealing key on Manage TrackMe Brokers. Queued requests and results not yet read then fail with sealed_unavailable, and a request already claimed carries on. Whether a failed call is tried again is up to its caller. Nothing changes on the add-on. The same action brings back a broker whose credential is truly gone.

Saving keys safely on the forwarder¶

A save is all or nothing. Splunk’s credential store takes a long value in 255-character pieces, one call each, so a save that stops part way would leave a document nobody can read, or one that reads and holds pieces of two. Each save writes a complete copy aside with its digest, then the document, then drops the copy.
One process per input. splunkd can start a second process while the first still runs, and two would overwrite each other’s keys. A file lock taken before the keys are read makes the second stand aside, and the system drops the lock however the process ends. Windows has no such lock, and there the next check is what ends an old process.
Removing a connection deletes its status file, and a running process notices that within one pull and stops. It also reads its input and its keys every minute, and stops when its input has gone or what is stored is no longer its own. So a connection added again under the same name starts with new keys.
A failure to read the keys never generates new ones: that would mean pairing again over a passing outage.

Revoking¶

Remove on a forwarder’s row, in Manage TrackMe Brokers, drops its keys and revokes the tokens TrackMe minted for it. Deleting a broker does both for each of its forwarders. The next call is refused, including a call held open, which reads the row again before it claims anything and every 15 seconds.
A forwarder refused with a proven instance_not_paired (removed, its broker deleted, or silent for 30 days) forgets its keys, generates new ones, and shows a new code. Its page reads removed_from_stack until it is paired again. A forwarder whose tokens were revoked is also refused with http_401, so it cannot register those keys until its connection holds a new token. One whose next call arrives after the revocation reads http_401 first, and hears this only then.
A forwarder rebuilt with no keys generates new ones and a new code. Accepting it replaces the instance’s row.
Removing a connection on the add-on’s page removes its private keys and does not reach the stack. What TrackMe minted for that forwarder stays valid until you select Remove on the stack, or until the stack drops that forwarder after 30 days of silence.
A token you entered yourself is yours to revoke, as with any Splunk token: TrackMe never revokes it.
Administrator actions are audited: an acceptance with its instance, fingerprint, account and source address, a removal with its instance, fingerprint, account and whether every minted token was revoked, a deletion with that same answer, and a change to a broker or a replaced sealing key. Each token TrackMe mints is logged in trackme_rest_api_broker.log, with its instance, account and id, and never the token.

The forwarder’s Splunk token¶

The stack issues this token, so TrackMe on the stack mints the next one, at the signed request of the paired forwarder. The role trackme_broker needs no extra capability, and a stolen token alone cannot renew itself, because renewal needs the forwarder’s signing key.

The order is the promise:

Mint. renew_token is signed by the forwarder’s current key: the previous key is refused. TrackMe calls POST /services/authorization/tokens on the system token, for the account pinned at pairing, never one the caller names, and it sets no expiry. The new token goes back sealed for this forwarder’s key, with the stack’s signature over this very call. The stack keeps token ids, never a token, in a row of their own, token:<broker>:<instance> in kv_trackme_broker_pairings, and not on the instance’s row, which only an acceptance or a rotation writes.
Prove. The forwarder makes confirm_token with the new token, so Splunk has just accepted it. It keeps the token in its key store from the moment it arrives, so a restart goes on from here.
Store. It writes the token into the connection, in the credential store on its side. Every thread reads it from there.
Retire. It makes confirm_token again, naming the token it replaces, and TrackMe revokes that one, along with anything else left on this forwarder’s record. It revokes only tokens this stack minted and still tracks for this forwarder, each under the account it was minted for. Any other id is left alone and reads as retired, so a signed call cannot revoke the token entered at setup or another forwarder’s.

A failure at any step leaves the token in use working. The next pass goes on from where this one stopped, an hour later, or at once on Renew token now. The connection shows the reason under Token renewal, and never reads as failing for it, because calls go on with the token in use. A token handed over and never confirmed is revoked by the next renew_token, unless the forwarder says that is the one it uses.

TrackMe mints on the stack’s system token. On Splunk Cloud this works with no capability added to the role: trackme_broker alone is enough. If Splunk ever refuses, the page reads token_mint_failed, the token in use keeps working, and it asks again every hour. Nothing else depends on renewal.

A minted token has the rights of the account pinned at pairing, which is one more reason for that account to hold trackme_broker and nothing else.

When it renews¶

When	Set by
Right after pairing, so each forwarder holds a token of its own	Renew the token automatically, on the connection
Every 7 days by default	Renew every (days), on the connection, from 1 to 365
When a token that states an expiry has under 30% of its life left	Renew the token automatically
On request	Renew token now, in the connection’s menu, once paired

Renew the token automatically turns off the first three, and Renew token now still works. With it off from the start, the forwarder goes on calling with the token you entered. Each forwarder of a pool renews on its own, and retires only tokens minted for it.

The token you entered¶

TrackMe never revokes a token you entered. Nobody here knows its id, and two forwarders may share it. It stays valid until you revoke it, and after the first renewal it is no longer used. So once Token renewed shows a time on every forwarder that used it, you can revoke it, and your token policy may well ask you to.

Removing a forwarder¶

Remove on the stack, or deleting its broker, revokes at once what TrackMe minted for that forwarder: the token in use, a token replaced and not yet retired, and tokens left from an earlier pairing. A call it holds open may hear first that it is no longer paired, and then its page also reads removed_from_stack. Any new call is refused with http_401, because its token is gone, so bringing it back takes a new token in its connection, and then a new code. A forwarder that never renewed still gets a new code by itself. After a Remove, the other forwarders of that broker keep their own tokens and carry on.

A forwarder silent for 30 days is dropped the same way, and what TrackMe minted for it is revoked, because a minted token has no expiry and nothing would name it again. So a forwarder that comes back after that also needs a new token in its connection.

A token that cannot be revoked at that moment stays on record, and the stack’s purge tries again until it is gone, because a minted token has no expiry of its own. Remove and Delete say whether everything was revoked, in the tokens_revoked field of their answer. Delete goes through every token record of that broker, so its answer also covers what an earlier Remove could not revoke.

When something is left, Manage TrackMe Brokers warns that a token is still valid and that TrackMe keeps trying, and the warning stays until the page reloads. The answer is audited, and trackme_rest_api_broker.log holds a warning. If that forwarder may be compromised, revoke its tokens in Splunk without waiting: their audience starts with TrackMe Broker.

Pairing a forwarder again moves the tokens of its earlier pairing to the new record, because one may be what it still calls with. The new record is written first and the old rows are deleted after it, so a write that fails loses sight of nothing. Each token keeps the account it was minted for, so one from a pairing under another Splunk account can still be revoked, and new tokens use the newly accepted account. These tokens go at its next retire call, and Remove and Delete revoke them too.

What the seal protects¶

The seal protects the hand-over. Once the forwarder calls with the new token, that token sits in the Authorization header like any other, and a TLS-inspecting proxy reads it. A token read that way still cannot pull, push or renew, and it goes at the next renewal.

This token is not a remote account’s token. See two independent tokens.

Cryptography in full¶

Why P-256 and not Curve25519¶

A forwarder with FIPS on runs cryptography under Splunk’s FIPS module, which refuses X25519, so sealing cannot use it. Ed25519 and ChaCha20 are refused the same way, and some Splunk Python builds refuse all three even without FIPS. Both sides must agree, so P-256 serves signing and sealing on both, with one curve throughout.

The sealing construction¶

Between the stack and a forwarder, each direction is sealed for the other side’s pinned key. It is an integrated encryption scheme: a fresh key on the sender’s side for every seal, against the recipient’s long-term key.

The sender generates an ephemeral P-256 key pair for this seal.
It runs ECDH between the ephemeral private key and the recipient’s public sealing key, which gives a 32-byte shared secret.
It derives a 32-byte key: HKDF-SHA256 with no salt and info = "trackme-broker-seal-v1\n" || ephemeral point || recipient point. Each point is 65 bytes, uncompressed. The info string binds the key to both public keys.
It generates a 12-byte random nonce, and encrypts with AES-256-GCM, using the associated data below. That gives the ciphertext and a 16-byte tag.
The sealed value is the base64 of ephemeral point || nonce || ciphertext || tag.

The recipient reads the ephemeral point, checks that it lies on the curve, and repeats the same ECDH and HKDF with its own private key. A public key that is not exactly 65 bytes, or not on the curve, is refused. Each seal has its own AES key, used once.

Seal	Plaintext	Associated data
A request, from the stack to the instance that claimed it	The envelope, as JSON	`<request_id>\n<claim_token>`: it cannot be replayed for another claim
A response chunk, from the add-on to the stack	`{body_b64}`, and `reason` and `headers` on chunk 0, as JSON	`<request_id>\n<claim_token>\n<chunk>`: it cannot be moved to another claim or position
A new Splunk token, from the stack to the instance that asked for it	The token	`trackme-broker-token-v1\n<broker>\n<instance_id>\n<ts>`: it opens only for that call of that instance

The associated data travels in clear beside the seal, and the receiver computes it from the fields of the call. So the seal proves that it belongs there.

A push wraps these layers: the body is gzipped, cut, base64-encoded, sealed as above, then placed in a JSON body, which the call’s signature covers, inside TLS. So the sealed value is what the signature covers: the order is seal, then sign.

Encryption at rest on the stack¶

Before anything reaches the KV store, the envelope and the response, its headers and body, are sealed with the broker’s own key. That is Fernet: AES-128-CBC with a random IV and HMAC-SHA256, through cryptography. The key lives in Splunk’s credential store on the stack. Each process caches it for a minute.

Row	Sealed
A request	`sealed_envelope`, removed when the request ends, and `sealed_meta`: the response’s reason and headers
A response chunk	`sealed_body`

In clear is what an operator reads and what runs the queue: broker, method, destination host and port, status, account, feature, caller, timing, failure code and message, deadline, attempts, claim token and retry_safe. A claim row holds nothing sealed. Field names never hold data.

The two seals are different on purpose. The channel’s seal is for the other side’s key, so the stack and the forwarder each read only what is for them. The at-rest seal is for the stack itself. The stack opens a request from its at-rest seal and seals it again for the instance at the claim, and opens a response chunk from the forwarder and seals it again at rest.

FIPS¶

TrackMe Broker is compatible with Splunk FIPS mode, on Splunk 9.4.x and later.

Every primitive here is FIPS-approved: ECDSA and ECDH on P-256, HKDF with SHA-256, AES-GCM, AES-CBC with HMAC-SHA256 for Fernet, and SHA-256. The broker’s own code asks for nothing else, on either side, and the add-on’s tests fail if its code asks for a primitive that FIPS does not approve. That is a property of the design and of the code. A FIPS validation belongs to a cryptographic module, and TrackMe Broker does not hold one of its own. What runs these algorithms differs on each side:

Side	What runs the algorithms	With FIPS on
Forwarder	`cryptography` from Splunk’s own Python, on Splunk’s OpenSSL	It runs under Splunk’s FIPS module, which is why X25519 is refused there.
Stack	The `cryptography` that TrackMe packages, with its own OpenSSL, which TrackMe’s handlers import before Splunk’s	It does not run under Splunk’s FIPS module. TrackMe uses the library it packages rather than Splunk’s module, with FIPS-approved algorithms only. No claim is made that this library is FIPS-validated. The choice of P-256 there is the design’s.

What the code does because of FIPS:

What it does	Why
One curve, P-256, for signing and sealing	X25519 and Ed25519 are refused on a forwarder with FIPS on.
Hashes the pairing code before it keys an HMAC	The code holds 100 bits, and a FIPS mode can refuse an HMAC key under 112 bits.
Uses `Cipher(AES, GCM)`, never the `AESGCM` class	That class copies a prepared cipher context for each call, and the FIPS provider of Splunk’s OpenSSL refuses the copy (`EVP_CIPHER_CTX_copy: not able to copy ctx`), for encryption and decryption alike. It is the same algorithm and the same bytes.
Guards splunklib’s list of TLS key-exchange groups, in the builds of TrackMe and of the add-on	On a connection that does not verify certificates, a FIPS provider can refuse that list, and every such connection would then fail before a packet is sent. A refused list leaves OpenSSL’s default groups.

Where a call can be read¶

Read from left to right: in the KV store a call is sealed with the broker’s key, and only what an operator reads and what runs the queue is in clear. TrackMe’s handlers hold it in clear in memory: the account’s credentials are TrackMe’s own. On the way to the forwarder it is sealed for the other side’s key and signed, inside TLS. The add-on holds it in clear in memory for as long as it carries it, and logs only its id, method, destination host and port, status and time. The target receives an HTTPS request from the forwarder’s address.

Destinations’ credentials stay in TrackMe and pass through the broker for each call. Nothing keeps a copy on the forwarder.

What a TLS-inspecting proxy sees¶

An inspecting proxy on the forwarder’s way out decrypts TLS, so it reads what sits inside it. Sealing is there for that case.

It can read	It cannot read
The Splunk token, in the `Authorization` header, a renewed one included as soon as the forwarder calls with it. See the stolen token below	The envelope: method, URL, headers, body
The channel’s path, the broker’s name and the instance id	The accounts’ credentials
What `pull` reports: the forwarder’s host, the add-on’s version, the digest and counts of its targets, its uptime	The response’s reason, headers and body
Request ids and claim tokens, attempts and seconds left, and the ids of tokens being renewed	The target list itself
The target’s HTTP status and time on a `push`, and the code and message of a `fail`. A `fail` with `target_not_allowed` says the host and port that no target covers	The private keys

It cannot substitute keys at pairing, because of the bindings. It cannot replay an answer, hand a forwarder another token, make a forwarder forget its keys or change its clock, because those are signed for one call. Check the certificate with the connection’s CA file: with verification on and the wrong CA, the connection fails and does not fall back.

TLS on each leg¶

Leg	TLS
Forwarder to stack	HTTPS. Verify the stack’s certificate is on by default, with an optional CA file on the forwarder. Off keeps HTTPS and no longer checks who answers. Signatures and seals still hold, since both sides pin their keys at pairing, but whoever answers receives the Splunk token.
Forwarder to target	HTTPS only: plain HTTP is refused. Verify the certificate and CA file belong to the target, on the forwarder, because only the forwarder sees that destination. The account’s own TLS options are not used. The forwarder reports a count of targets without verification, never the list.
TrackMe to its local splunkd	The exchange endpoints, on the stack’s own splunkd, never through a proxy. TrackMe calls them without verifying the certificate, carrying the session key.

Threat model¶

Threat	Outcome
A stolen Splunk token	It cannot pull or push, and a token cannot act for a forwarder without that forwarder’s signing key. It can register pending rows, which an administrator would have to accept, up to 20 for its account. It can tell brokers and instances apart from the refusals, read the stack’s time, and make a code ambiguous: see the limits below. Its role is built to reach the channel and nothing else. It cannot renew itself, because `renew_token` and `confirm_token` are signed channel calls. A token TrackMe minted goes at the next renewal. A token you entered is yours to revoke.
A registration by someone holding a token	It needs its own code, so an administrator must be given that code. The fingerprint, source address and account are shown first. On Splunk Cloud, the source address is the load balancer’s and not your forwarder’s, so the fingerprint is what to compare.
A stolen pairing code alone	Useless without a token, and gone in 15 minutes. A row holds a binding, not the code.
A stolen pairing code and a token	Registering under it makes the code ambiguous, which accepts nobody and tells the administrator. If the forwarder’s own row has run out, the preview shows the thief’s fingerprint, which does not match the add-on’s page, and an acceptance lands only on the row previewed.
TLS inspection on the way out	It sees signed, sealed values and the metadata above. It cannot substitute keys, replay an answer, or change what a forwarder holds.
A replayed answer to a `pull`	Refused: the signature covers the nonce of the pull that asked.
A replayed call to the stack	Bound by the 300-second window. See the limits below.
A forged refusal, or a forged answer to a rotation or a renewal	Ignored: only the pinned stack key, signing for this very call, changes what a forwarder holds.
A replayed `renew_token`	Refused as `stale_call`, and nothing is minted: `ts` must grow per instance, and a short-lived row where the first insert wins stops a copy arriving alongside the original. The new token travels sealed for that forwarder’s key, signed for that very call.
Renewal turned on another account	It cannot be: TrackMe mints for the account pinned at pairing, never one the caller names, and the call must come from that account’s token.
A signed `confirm_token` naming a token that is not its own	Left alone, and it reads as retired. TrackMe revokes only tokens this stack minted and still tracks for that forwarder, each under the account it was minted for. So a signed call cannot revoke the token entered at setup or another forwarder’s.
A compromised stack, or a stack administrator acting against you	The forwarder’s target list bounds them. They can ask a declared target for anything its origin or path allows, with any method, and read the answer. They cannot reach an address no target declares, change a target, turn verification on or off, read a private key of the forwarder, or read the target list, though a call to a guessed destination shows whether a target covers it. No code from the stack runs on the forwarder: it makes HTTP calls. Every call is logged on the forwarder itself, with its destination, in `trackme_broker.log`. A path on a target narrows what it allows. Splunk and Cribl clients need the whole origin.
Someone reading KV rows	They read ciphertext, and what an operator reads and what runs the queue. A key’s row and a token’s row hold ids, never a secret. Whoever can also read the stack’s credential store holds the keys: this does not protect against that.
Someone reading a TrackMe backup	Backups hold the list of brokers and no key. They leave out instances, requests, claims, responses, pairings and heartbeats. So after a restore onto a rebuilt deployment, broker definitions come back with no keys, pairings or queues, and calls are refused with `broker_offline` until a forwarder is paired. Each forwarder keeps its keys, and its page says to remove the connection and add it again (`not_paired_unverified`). Accepting the new code into the restored broker’s name works, and the first `enqueue` after that creates a new sealing key.
A forwarder’s private key stolen	Full compromise of that instance, including what it carries: requests hold the destination’s credentials. With a token for its account, it can also ask for new tokens, for that account only. Remove it on the stack, which revokes those too. The target list still bounds what it can reach, and the keys of the other instances are their own.
DNS changing its answer	The targets screen what a name answers, and the connection resolves it again. So the add-on checks the address the connection actually reached, before TLS and before anything is sent: a name that answers differently once connected fails with `target_not_allowed`, with nothing sent. A name that answers with another allowed address is not refused: a target declares a name, not an address, so TLS verification is what says who answered.
A TrackMe user turning the broker on something else	`enqueue` is authorised against the account: bound to this broker, a destination the account declares, a caller the account allows. An exchange belongs to the user who queued it.
A URL or a header built to walk past a target	Two parsers must agree, encoded paths are decoded before they are judged, `Host` and framing headers are dropped on both sides, and the forwarder does not follow redirects.
A target’s instance metadata	Always refused, by name and by address, whatever a target says.
Plain HTTP	Refused: at the envelope on the stack, and again by the targets on the forwarder.

Known limits¶

What the design does not cover, stated plainly.

Replays towards the stack stay inside the 300-second window. Apart from renew_token, the stack does not remember the calls it has seen. A captured, signed pull replayed within that window claims real requests for an answer that the real instance never sees. Whoever replays it cannot open them, since they are sealed for the instance. Those requests end as a lost lease: a call safe to repeat goes back to the queue, and any other fails with broker_lost. An actor on that path could stop the traffic anyway. A replayed push, progress or fail names a claim, and is idempotent or refused. A replayed confirm_token answers the same way, and a replayed renew_token is refused and mints nothing.
A token holder can block a pairing. Repeating a registration, or registering under a stolen code, makes that code ambiguous. Nobody is accepted on the wrong keys, and the administrator sees the refusal. The add-on’s next code, within 15 minutes, starts clean, though the same holder can repeat this while they hold the token. Revoke the token.
A seal has no forward secrecy against its recipient’s key. The sender’s side is ephemeral and the recipient’s is long-lived. Whoever records traffic, and later steals a recipient’s sealing key, can open what was sealed for that key. An instance’s sealing key rotates every 30 days, and the add-on does not keep retired keys in storage. The stack’s sealing key does not rotate. TLS underneath, if nothing inspects it, normally has its own forward secrecy.
At-rest sealing protects KV rows, not the stack. The broker’s key and the stack’s keys sit in the stack’s credential store. An account that can read it can open what is sealed there.
The stack’s keys do not rotate, and replacing them means pairing again. Nothing on Manage TrackMe Brokers does it. A leaked stack key is answered by deleting its row and pairing every forwarder again: see keys that live longer.
A call’s checks stop at the host and port. enqueue compares the destination’s host and port with what the account declares. The method, path, headers and body are the caller’s. Anyone holding trackmeuseroperations and the rights on an account can call it directly, and send anything to that destination that the forwarder’s targets allow.
Two users pass every rights check. splunk-system-user and admin pass the rights check of every account.
A token alone is not quite nothing. From the refusals it can tell brokers and instances apart. With the account pinned for a forwarder, it can read the stack’s time from a clock_skew. It can fill the 20 pending rows of its account, which stops new registrations for that account until they run out.
The target digest can be tested offline. It is an unsalted SHA-256 of the sorted URLs, truncated to 64 bits, so whoever reads it can test a candidate list. It covers URLs only: forwarders with the same URLs and different certificate settings report the same digest.
A caller on the stack can follow a redirect. The forwarder does not follow one: a 3xx goes back as the answer. A requests caller on the stack that follows redirects then does so itself, as a new exchange under the same checks.
TrackMe trusts its local splunkd. It calls the exchange endpoints without verifying the certificate, carrying the session key.
The fingerprint is for a person to compare. It shows 64 bits of a SHA-256. A person who does not compare it adds nothing. The binding, not the fingerprint, is what stops keys being substituted on the way.
The keys on the forwarder are as safe as the forwarder. An administrator of that Splunk instance can read its credential store.
A rotation can race an acceptance. A rotation reads its row again just before writing, so a re-pairing that landed in between is not undone. The KV store has no compare-and-swap, so one landing within that very round trip still could be. A rotation comes every 30 days, and an acceptance is an administrator’s act, so this is accepted.
A connection removed on the add-on’s page is not told to the stack. What TrackMe minted for that forwarder stays valid until you select Remove on the stack, or until the stack drops that forwarder after 30 days of silence and revokes it. Its audience names the forwarder, TrackMe Broker <broker> <instance>, so an administrator can revoke it sooner in Splunk.
A token you entered, and renewal that does not run, stay yours to manage. TrackMe never revokes a token you entered. With renewal off, or refused by Splunk, the token in use goes on until it expires or you replace it.