TrackMe Broker — in depth¶

Tip

This is the in-depth reference for how a call travels through a broker. For the concept and the setup, start with TrackMe Broker. For pairing, keys, ciphers and the threat model, see TrackMe Broker — security and cryptography.

The transport¶

An account’s connectivity field holds direct, the default, or broker:<name>. An absent or empty field is direct, and that path is unchanged. Anything else that is not broker:<name> raises invalid_connectivity: a mistyped field never falls back to a direct connection that nobody asked for.

For a broker, one piece is swapped under the client and the calling code stays as it is:

Client	What is swapped	A failure raises
`requests` (Cribl connections, a remote account’s REST calls, token rotation, account tests)	A transport adapter, mounted on the session for `https://` and `http://`. A plain URL then fails at the envelope, and never leaves.	`Timeout` for `deadline_exceeded` and `target_timeout`, `SSLError` for `tls_verification_failed`, `ConnectionError` for the others
splunklib (`splunkremotesearch`, other SDK calls)	The `handler` of the connect call	`ConnectionError`
Outbound HTTP client (webhook accounts)	Its `transport`	The client’s own outcomes: delivery unconfirmed, not sent, or a final refusal

The failure is what the client would have raised on a direct connection, with the broker’s code in its text. A caller’s retries, its failover between endpoints and its error reporting keep working unchanged.

Who may send a call¶

The exchange endpoints, POST /services/trackme/v2/broker/exchange/<enqueue|result|result_chunk|release|cancel>, require the capability trackmeuseroperations. TrackMe’s code calls them on the local splunkd, and they are not exposed through Splunk Web. enqueue then checks the call against the account it is made for. The rights come first: the caller must be allowed to use the account, as a direct call would require, so someone without those rights learns nothing about its broker or destination from the answer. Then the account must be bound to this broker, and the destination must be one the account declares, by host and port. The known limits say what that leaves to the caller.

Account type	Who may use it
`remote_account`	A user holding one of the account’s roles
`cribl_connection`	TrackMe administrators, as on the direct path
`webhook_account`	A power user or an administrator the account allows. The rights are checked before the destination, so nobody can test a guess at a secret URL’s host
`account_test`	TrackMe administrators, for a test before an account exists. The forwarder’s targets still decide what is reached

The users splunk-system-user and admin pass each of these rights checks. Only the user who queued an exchange can read it. For anyone else it reads as absent.

Where things live on the stack¶

The KV store is where the process that wants a call made meets the process that answers the forwarder, which can be another search head. Handlers write with the system token, after their own checks. The KV store only replaces whole documents, so what changes often sits apart from what must never be overwritten.

Collection	Key	What it holds
`kv_trackme_brokers`	Broker name	Description, enabled
`kv_trackme_broker_instances`	`<broker>:<instance>`	What pairing pinned: both public keys, fingerprint, Splunk account. Written only by an acceptance or a rotation
`kv_trackme_broker_heartbeats`	`<broker>:<instance>`	What each pull reports: last seen, host, version, protocol, concurrency, target digest and counts, uptime, source address
`kv_trackme_broker_requests`	Request id	`sealed_envelope`, `sealed_meta`, status, deadline, attempts, claim token. In clear for an operator: broker, method, destination host and port, account, feature, caller, HTTP status, time at the target, failure code
`kv_trackme_broker_claims`	`<id>:<attempt>`	Who claimed it, and its lease. Also `<id>:lost:<attempt>`, the decision on a lost lease
`kv_trackme_broker_responses`	`<id>:<attempt>:<chunk>`	`sealed_body`: one chunk. By attempt, so a late push of an old attempt can never mix with the next one’s chunks
`kv_trackme_broker_pairings`	Registration id, `key:<name>`, `token:<broker>:<instance>` or `token_asked:<broker>:<instance>:<ts>`	Pending and accepted registrations, see pairing. The row of each key, which names its credential, see keys on a search head cluster. The ids of the tokens TrackMe minted for an instance, never a token, see the forwarder’s token. A short-lived row for each `renew_token`, where the first insert wins, so that a copy of the call mints nothing

The channel¶

Every call comes from the forwarder, as POST /services/trackme/v2/broker/channel/<operation> on the stack’s management port, with a bearer token for an account holding trackmebrokeroperations. Bodies are JSON. The add-on reads anything but a 200 as a refusal, redirects included, and never follows one.

Operation	Signed	What it does
`register`	By its own new key	Offers this instance’s public keys, with a binding to the pairing code. Stores a pending row, and nothing else
`pairing_status`	By its own new key	Asks where a registration stands. Answers `pending`, `expired` or `accepted`, the last with the stack’s keys
`pull`	Yes	A heartbeat, and a long poll, which Splunk Cloud’s load balancer does not cut. Answers with the requests it claimed for this instance
`progress`	Yes	Extends the lease of a request still running. A renewal is also a sign of life
`push`	Yes	Delivers one chunk of a response. The last chunk completes the exchange
`fail`	Yes	Reports that a request produced no HTTP response, with a fixed code
`rotate_key`	Yes	Replaces this instance’s keys, see rotation
`renew_token`	Yes, current key only	Asks for a new Splunk token for the account pinned at pairing, see the forwarder’s token
`confirm_token`	Yes	Made with the new token, which proves that it works. Made again naming the token it replaces. TrackMe revokes only what it minted and still tracks for this instance: any other id is left alone and reads as retired

A signed call wraps its body as a string, so that nothing has to be canonicalised on the way:

{"broker": "onprem", "instance_id": "forwarder-a:main", "ts": 1793776421,
 "body": "{\"max_requests\":4,\"wait_seconds\":20,\"nonce\":\"...\"}", "signature": "..."}

An instance id is the forwarder’s host name and the connection’s name.

Protocol versions¶

Both sides speak protocol 1. A pull carries the add-on’s protocol under instance, and a stack that does not support it refuses with protocol_unsupported (HTTP 426). The two sides ship separately, so anything in this contract changes only with a new protocol number.

The life of one call¶

Queued on the stack¶

The transport builds the envelope.

Field	Meaning
`method`, `url`	As the caller built them. HTTPS only, with no credentials and no fragment in the URL. The method is one of GET, HEAD, POST, PUT, PATCH, DELETE or OPTIONS
`headers`	A list of `[name, value]` pairs. Hop-by-hop headers and `Host` are dropped on both sides: what the forwarder’s own client decides cannot be contradicted, so a caller cannot reach an undeclared virtual host behind a shared front end
`body_b64`	The body, base64, at most 8 MiB decoded. It is not chunked: a larger body is refused at once with `request_too_large`
`timeout_seconds`, `connect_timeout`, `read_timeout`	The budget, see Timeouts
`retry_safe`	Whether repeating the call is harmless. It decides what a lost lease does. By default, true for GET, HEAD and OPTIONS. A caller may say otherwise
`response_limit`	Bytes the forwarder may return: 64 MiB by default, and 256 MiB for an SDK call, because a remote search’s results are one HTTP response and a direct connection has no limit on it. The stack refuses more than 256 MiB. The add-on applies the lower of this and its own limit

TLS is not in the envelope. Verification and a private CA belong to the target declared on the forwarder, because only the forwarder sees that destination.

enqueue checks the envelope and the caller’s rights on the account. It then refuses at once what cannot be served: broker_disabled, broker_offline when no instance has been seen for 90 seconds, broker_busy when 500 requests are queued.
The envelope is encrypted with the broker’s key and written as a queued row, with its deadline. The caller receives a request id.

Handed over at the forwarder’s pull¶

The add-on holds one signed pull open at a time, with a fresh nonce, while it has room for more work:
```
{"max_requests": 4, "wait_seconds": 20, "nonce": "32 hex characters, fresh for this pull",
 "instance": {"host": "forwarder-a", "add_on_version": "1.1.1", "protocol": 1, "max_in_flight": 4,
              "allowlist_digest": "...", "targets": 3, "targets_unverified": 1, "uptime_seconds": 120}}
```
The stack holds it for up to wait_seconds, at most 25, and answers as soon as work exists. An empty answer ends the hold, and the add-on pulls again. If a stack answers an empty pull at once, the add-on still waits a second before the next one. While it holds a call, the stack looks at the queue every 0.25 seconds, and every 15 seconds it checks that this instance is still paired, looks for ended leases and records a heartbeat.
When work exists, the stack first reads everything a claim depends on: that the instance is still paired, and the stack’s keys. If the credential store does not answer, no request is lost.
A claim is an insert of <request id>:<attempt> in kv_trackme_broker_claims: the first inserter wins, on any search head. The row holds a lease of 60 seconds.
The stack opens the envelope with the broker’s key, and hands it over sealed for the key of the instance that claimed it, with the request id and a new claim token as associated data. Each instance of a pool gets its own copy. The stack signs the seal, the claim, the attempt, the seconds left and the nonce of this very pull:
```
{"protocol_supported": [1], "requests": [
  {"request_id": "...", "claim_token": "...", "lease_until": 1793776481, "remaining_seconds": 58, "attempt": 0,
   "sealed": "...", "signature": "..."}]}
```
One pull claims at most 20 requests, and stops adding once it has handed over 16 MiB.

Verified and carried out by the add-on¶

The signature is checked first, for the nonce it sent, before the request takes a slot or a lease renewal. One that does not verify gets no answer, because a fail signed for it would end whatever claim it names: its lease runs out, and the add-on’s page says that requests arrived that TrackMe’s key did not sign. A claim runs once, whatever an answer holds.
The seal is opened. One the stack signed and that nothing here can open answers invalid_envelope at once.
The targets decide. The URL must pass the checks below, and a declared target must cover it. Otherwise the call ends as target_not_allowed with nothing sent.
The executor connects, and checks the address it actually reached, before TLS and before anything is sent. The target’s certificate settings apply. Redirects are not followed: a 3xx goes back to the caller as the answer.
Every 20 seconds while it runs, the add-on sends progress, which extends the lease by 60 seconds. A saturated instance stops pulling, and these renewals also say that it is alive. A progress or a push on a claim that has ended, a cancelled exchange for one, is refused with claim_mismatch, and nothing more from that attempt is stored. The call already sent to the target is not recalled.

Which destinations are allowed¶

The add-on applies these rules, and the stack has no say in them.

A target is https://host[:port][/path]. The port defaults to 443 and must match. A wildcard *.example.com covers subdomains and not example.com itself. A path covers itself and what sits under it, on a segment boundary.
A URL with credentials, a backslash or a control character is refused. Two parsers must agree on the host and port: the one that checks and the one that connects.
A path cannot be walked out of. It is decoded for as long as decoding changes it, up to five rounds, and a . or .. segment is refused, as is a path that still decodes after that.
Private addresses are allowed, since a target declares them, and so is loopback: https://localhost:8089 is the forwarder’s own management port, which only suits a broker with one forwarder.
Some addresses are always refused, whatever a target says, for what the name answers and again for the address a connection reaches: link-local (169.254.0.0/16, fe80::/10), the instance metadata addresses (fd00:ec2::254, 100.100.100.200, 168.63.129.16), multicast, unspecified and reserved addresses. An IPv4-mapped or NAT64 address counts as the IPv4 address it stands for. The names metadata.google.internal, metadata.goog, instance-data and instance-data.ec2.internal are refused by name.
Ambient proxies and .netrc credentials are never used for a target.

Answered in sealed chunks¶

The add-on reads the response and gzips it as it reads. A body above the limit stops there with response_too_large, and so does one that would need more than 256 chunks. Hop-by-hop headers, Content-Encoding and Content-Length are dropped, since they no longer hold.
The gzipped body is cut into pieces of 768 KiB, each 1 MiB once in base64. Each piece is sealed for the stack’s key, with the request id, claim token and chunk number as associated data. Chunk 0 also holds the reason and the headers, in the same seal, and carries the status code beside it.
push delivers the chunks in order. The stack opens each one, encrypts it again with the broker’s key, and writes it. A repeated chunk replaces itself, so a push can be retried: up to 5 attempts for each report through a short outage. The last chunk marks the request done, if all the chunks and the headers are present. Otherwise the stack answers incomplete_response.
If there was no HTTP response, the add-on sends fail, with a fixed code, and the request ends as failed.

Read back by the caller¶

The caller polls result, each poll held up to 5 seconds. A poll is also how an ended lease or a queue nothing can serve is noticed, when no pull is doing it.
Once the status is done, the caller reads each chunk through result_chunk. A chunk stays readable until the release, so a read can be repeated: a target that already ran the call must not read as a failure because an answer was lost on its way back.
The caller joins the chunks, gunzips them under the same limit, and rebuilds what its client expects.
release removes the request, its claims and its chunks.

Timeouts¶

A call has one budget. It is stamped once, at enqueue, as deadline = now + timeout_seconds, on the stack’s clock. Time in the queue counts.

How a budget gets into the envelope¶

What the client asked for	`timeout_seconds`	`connect_timeout`	`read_timeout`
`requests` with `timeout=(connect, read)`	connect + read	connect	read
`requests` with `timeout=t`	2 × t	t	t
`requests` with no timeout	60 s	30 s	60 s
A splunklib call, through `client.connect`	3600 s	30 s	3600 s
An SDK transport built with its own limit `b`	b	the lower of b and 30 s	b
The webhook client	connect + read, or what its own deadline leaves if that is less	connect	read
Test in Manage TrackMe Brokers	30 s	30 s	30 s

Whatever the source, the transport keeps timeout_seconds between 1 and 3600 seconds, and the stack refuses a value above 3600 seconds in any of the three fields.

The search timeout of a remote account¶

A remote account’s timeout_search_check, 600 seconds by default, is passed to splunklib’s connect call as timeout. splunklib ignores that argument, on a direct account too: on a direct connection, one SDK call has no time limit.

A brokered SDK call does have one, because an exchange must. It takes the channel’s maximum, 3600 seconds, so that a long export does not fail only because it goes through a broker. timeout_search_check does not shorten it. A TrackMe component that needs a tighter bound builds its own SDK transport with that bound, and the same bound then travels through the broker.

timeout_connect_check, 15 seconds by default, picks a reachable URL among an account’s endpoints, through requests with a single value. Through a broker it therefore gives connect_timeout and read_timeout of 15 seconds, and timeout_seconds of 30.

How the forwarder uses it¶

The forwarder budgets from how long is left, not from a time of day. A claim hands over remaining_seconds as an integer that the stack’s signature covers, and the forwarder adds it to its own clock, because the two clocks need not agree. A hand-over also carries lease_until, an absolute time on the stack’s clock: it is not signed, and the add-on ignores it. Then:

Both timeouts are computed once, from what remains when the call starts. The connection gets the lower of connect_timeout and what remains, and each read gets the lower of read_timeout and what remains. The read timeout is the longest silence between two reads, not a limit on the whole body.
So a short budget caps both, and the code follows the stage rather than which limit was reached: a budget that runs out while connecting reads as target_unreachable, and one that runs out while waiting for the answer reads as target_timeout.
While the body arrives, the deadline is also checked between blocks of the body. Past it, the call ends as deadline_exceeded.
If nothing remains when the call starts, it ends as deadline_exceeded without connecting.

The caller waits until the deadline plus 5 seconds, because an answer in flight when its time ends costs nothing to wait for, and the alternative is to report a failure for a call that worked. Then it cancels, and raises deadline_exceeded. The deadline is on the row as well, so the stack expires it even if the caller has gone.

Which code means what¶

Code	When	Could the call have reached the target?
`target_unreachable`	The connection never opened: refused, no route, a name that does not resolve, or `connect_timeout` reached, or the budget if that is less. This is a promise that nothing was sent, and TrackMe may send the call again on that word.	No
`tls_verification_failed`	The certificate was refused, at the handshake.	No
`target_timeout`	The target did not answer within `read_timeout`, or within the budget if that is less.	Yes
`target_aborted`	The connection was lost after the request may have been written, or the body was cut short. A `read_timeout` reached while the body arrives reads like this, and so does a TLS error other than a refused certificate.	Yes
`deadline_exceeded`	The budget ran out anywhere else: before the call started, while the body arrives, at a claim, at the caller’s poll, or at the caller’s own limit.	Yes

Pools, leases and failure¶

A broker on the stack is an ensemble. An account binds to the broker’s name and never to a host, and each forwarder paired to that broker is one instance. Every instance pulls from the same queue. There is nothing to elect and no failover step: this is active/active, with no delay on new calls, and both forwarders share the load.

A request is queued, then claimed, then one of done, failed, expired or cancelled. The one way back is a lost lease on a call that is safe to repeat.

When a forwarder stops¶

A lease ends 60 seconds after the claim or the last progress. The stack looks for ended leases at the start of a pull, every 15 seconds while it holds one, and when a caller polls. The decision is taken once, through an insert of <id>:lost:<attempt>, because a sweep on one search head and a poll on another can both see the same ended lease, and a second write would undo the claim that followed the first.

The call	What happens
New	The surviving instance takes it at its next pull.
Running, safe to repeat	Back in the queue when its lease ends, with the same deadline, and the survivor runs it as a new attempt with its own claim token. At most 3 attempts. The target may see it more than once, by design.
Running, not safe to repeat	Fails with `broker_lost`, because the target may have acted on it. The caller keeps its own retry rules.
Safe to repeat, with its 3 attempts used	Fails with `broker_lost` as well.
Past its deadline	Whichever comes first decides. A sweep gives `broker_lost`, even for a call that is safe to repeat. A caller’s poll gives `deadline_exceeded`, even for one that is not.

A late push from an attempt that lost its lease is refused with claim_mismatch. Chunks are stored by attempt, so they can never mix.

When nothing can serve the queue¶

A pull writes a heartbeat at most every 15 seconds, and a progress does as well, so a saturated instance reads as alive. An instance not seen for 90 seconds is offline. With no instance online:

enqueue refuses at once with broker_offline;
a request already queued fails with broker_offline at the caller’s next poll, rather than at a deadline that can be an hour away.

A disabled broker behaves the same way with broker_disabled, and also refuses its forwarders’ calls, including one held open. An enqueue on a deleted broker is refused with unknown_broker. Its queued requests went with it, so a caller polling one gets unknown_request.

What Configuration Guardian raises¶

broker_offline warns when no forwarder of a broker has pulled for a while, then turns critical. broker_degraded warns when a forwarder stopped while others carry on, or when their target lists differ. Configuration Guardian alerts gives what each says, what it recommends, when it clears and its settings.

These checks use their own settings. Online on Manage TrackMe Brokers means seen in the last 90 seconds, so the page shows a stopped forwarder before the alert does.

Stopping and removing¶

Disable a connection on the add-on’s page: its process reads its input every minute, and stops. Calls it was running go back to the queue if they are safe to repeat, and fail otherwise.
Remove a connection on the add-on’s page: its keys go, its process notices within one pull and stops, and its line leaves the page. Calls it was running end the same way. Added back under the same name, it shows a new code within about a minute, or at splunkd’s next interval if that is later.
Remove a forwarder in Manage TrackMe Brokers: its key is dropped, TrackMe revokes what it minted for that forwarder at once, and its next call is refused, including a call held open, which reads the row again. With its token revoked, it reads http_401, so it needs a new token in its connection before it can show a new code. If a call held open hears the stack’s refusal first, the add-on also verifies it, forgets its keys and reads removed_from_stack. A forwarder that never renewed shows a new code directly. Other forwarders of the broker carry on.
An instance silent for 30 days is removed from the broker in the same way, and has to pair again.
Delete a broker: its forwarders, its queue and its key go, and TrackMe revokes what it minted for those forwarders. Accounts bound to it fail with unknown_broker until you change their Connectivity.

Failure codes¶

Fixed strings, never a value, a body or a secret.

Code	Set by	Meaning
`target_not_allowed`	Add-on	The URL is not among the forwarder’s targets, or the address it reached is one always refused
`target_unreachable`	Add-on	The connection never opened, so nothing was sent
`target_aborted`	Add-on	The connection was lost after the request may have been written, or the body was cut short
`target_timeout`	Add-on	The target did not answer in time
`tls_verification_failed`	Add-on	The target’s certificate was refused, at the handshake, so nothing was sent
`response_too_large`	Add-on	The response is above the limit, or would need more than 256 chunks
`request_too_large`	Stack	The request body is above 8 MiB. Refused at `enqueue`, so nothing was queued
`deadline_exceeded`	Either	The budget ran out
`invalid_envelope`	Either	The stack refused the envelope at `enqueue`. Or the stack signed this request and the add-on cannot open it, or its executor cannot use what it holds
`broker_error`	Add-on	An unexpected failure in the add-on
`broker_lost`	Stack	An instance lost its lease on a call that cannot be repeated
`broker_offline`	Stack	No instance seen in the last 90 seconds
`broker_busy`	Stack	The queue is full: 500 requests
`broker_disabled`, `unknown_broker`	Stack	As named
`cancelled`	Stack	The caller cancelled the exchange
`sealed_unavailable`	Stack	The broker’s key cannot open the row: it was replaced, or it is gone. A credential store that does not answer is not this, and nor is a key still on its way (`key_replicating`, below): the row stays queued for the next pull
`claim_mismatch`	Stack	This claim no longer owns the request
`broker_user_refused`	Stack	The token’s account is not the one pinned when this forwarder was accepted
`instance_not_paired`, `invalid_signature`, `clock_skew`	Stack	See the signed channel
`pairing_code_ambiguous`, `pairing_changed`, `pairing_code_not_found`	Stack	More than one registration claims this code; the row previewed is no longer the one this code finds; none does
`account_not_bound_to_broker`, `destination_not_of_account`, `account_access_denied`	Stack	The caller’s account does not allow this exchange

What the endpoints answer¶

These are refusals of one call to an endpoint, rather than how an exchange ends.

TrackMe’s caller, on the exchange endpoints	Meaning
`invalid_request`, `invalid_broker_name`, `invalid_request_id`	The call itself is malformed
`invalid_url`	Not an HTTPS URL, or it holds credentials or a fragment
`invalid_connectivity`	The account’s `connectivity` is neither `direct` nor `broker:<name>`
`unknown_account`, `unknown_account_type`	No such account, or no such type of account
`unknown_request`	No such exchange, one queued by another user, or one whose broker was deleted
`response_not_ready`	A chunk was asked for before the exchange was done
`invalid_chunk`, `unknown_chunk`	A chunk number out of range, or a chunk not stored
`key_replicating`	A key decided on another search head has not reached this one, see keys on a search head cluster. Nothing changed. A poll tries again by itself. An `enqueue` was not queued, and asks again for as long as its own time lasts
`broker_unavailable`	A passing failure. A poll tries again by itself. An `enqueue` that gets this may have been queued

A forwarder, on the channel	Meaning
`invalid_request`, `invalid_instance_id`, `invalid_request_id`	The call itself is malformed
`too_many_pending_pairings`	This account already holds 20 pending registrations
`protocol_unsupported`	The stack does not support the protocol this add-on speaks
`invalid_chunk`, `chunk_too_large`, `invalid_status_code`	A `push` out of range, with a chunk above the limits, or with a status outside 100 to 599
`incomplete_response`	The last chunk arrived with chunks or headers missing
`key_replicating`	A key decided on another search head has not reached this one. Nothing changed, and the add-on backs off. A `push` or a `fail` is retried until it lands or its claim ends
`token_mint_failed`	Splunk refused to mint a token on the system token. The token in use stands
`token_changed`	The token named is neither the pending one nor the current one. The token in use stands
`stale_call`	A `renew_token` played again: its `ts` is not newer than the last one, or a copy arrived alongside the original. Nothing is minted
`broker_channel_unavailable`	A passing failure on the stack. The add-on tries again

An administrator, on Manage TrackMe Brokers	Meaning
`invalid_code`	This cannot be a pairing code: wrong length, or a character outside the alphabet
`invalid_enabled`	A value that cannot be read
`broker_exists`	A broker of that name was created at the same moment
`unknown_instance`	That forwarder is no longer on this broker
`broker_list_truncated`	More than 1000 rows, so the page refuses to show part of a list
`key_replicating`	TrackMe’s keys have not reached this search head yet, and nothing has changed. Add broker asks again by itself and says so
`broker_admin_unavailable`	A passing failure. Try again

What a failure proves¶

A caller that retries a POST on the word “not sent” must be able to trust it. So that word follows the stage of the exchange, and not the code alone. By default it is false: not knowing is not the same as not sent.

What happened	Known not to have left?
The stack answered the `enqueue` with its own refusal	Yes
The `enqueue` got no answer, or a proxy’s error page	No: it may have been queued
A poll failed after the `enqueue`, whatever its code	No
The exchange ended `failed` with `target_not_allowed`, `target_unreachable`, `tls_verification_failed`, `invalid_envelope`, `broker_offline`, `broker_disabled`, `unknown_broker` or `sealed_unavailable`	Yes
The exchange ended any other way	No

The webhook client turns this into its own outcomes: not sent is worth another try, and anything not proven reads as delivery unconfirmed.

Operating a broker¶

Where to look¶

index=_internal sourcetype=trackme:broker

This is the add-on’s log, from the forwarder. A state line says why it cannot reach the stack, such as broker channel state="disconnected:http_401", with one line when the state changes and not one per attempt. Each call has one line, with its id, method, destination host and port, status or failure code, time, bytes and chunks. On the stack, the handlers log to trackme_rest_api_broker.log.

Manage TrackMe Brokers shows each broker’s Exchanges in flight and recent failures. Failures stay readable for a day.

States on Manage TrackMe Brokers¶

Manage TrackMe Brokers gives each state and what a forwarder’s row shows.

States and reasons on the add-on’s page¶

Status reads Starting, Waiting to be accepted, Connected, Disconnected, Disabled, Stopped, Not running or Unavailable. A connection also says why, as a sentence. These are the reasons behind those sentences, as the add-on’s status file and log name them. In the log, unreachable comes with the exception’s type, such as unreachable:SSLError, and any other refusal without a code of its own as http_<status>.

Reason	What to do
`http_401`	The stack refused the token. Check that it has not expired or been revoked. A forwarder removed on the stack, or silent for 30 days, had what TrackMe minted for it revoked: enter a new token in its connection, and it then shows a new code
`http_403`	The account behind the token may not use the channel. It needs the `trackme_broker` role
`http_404`	The stack has no TrackMe channel. Check the URL, and TrackMe’s version there
`unreachable`	Check the URL, the network, and that the `search-api` allow list includes this forwarder
`broker_disabled`	The broker is disabled in TrackMe
`broker_user_refused`	TrackMe paired this forwarder with another Splunk account. Use that account’s token, or pair again
`invalid_signature`	TrackMe did not recognise this forwarder’s key. Remove the instance in TrackMe, and pair again
`protocol_unsupported`	The TrackMe version on this stack does not support this add-on version
`removed_from_stack`	TrackMe no longer knows this forwarder, so it made new keys. Add it in TrackMe with the code shown. If the connection also reads `http_401`, its token was revoked too: enter a new one first
`not_paired_unverified`	Something answered that this forwarder is not paired, without the stack’s signature, so the keys are kept. If the stack was rebuilt, remove this connection and add it again
`request_signature_invalid`	Requests arrived that TrackMe’s key did not sign, and none was carried out. If this goes on, remove the instance in TrackMe and pair again
`rotation_not_confirmed`	Something answered a key rotation without the stack’s signature, so the keys did not change
`key_replicating`	The search head this reached has not received TrackMe’s keys from the others yet. This clears by itself once it does. If it never does, see keys on a search head cluster
`acceptance_not_bound_to_code`	An acceptance did not match this pairing code, and was refused. Check who accepted this broker
`cryptography_missing`	This Splunk’s Python has no `cryptography` library, so nothing can be signed or sealed

A renewal that does not finish has its own messages, under Token renewal in the connection’s details. It never reads as the connection failing, because calls go on with the token in use.

Reason	What it means
`token_mint_failed`	TrackMe could not mint a new token on the stack. The token in use keeps working, and this is tried again every hour
`token_not_confirmed`	Something answered a token renewal without the stack’s signature, so the token was not changed
`token_not_retired`	The new token is in use, but TrackMe could not revoke the one it replaces yet. This is tried again every hour
`token_changed`	The stack no longer knows the new token, so the token in use stays. This is tried again every hour
`stale_call`	The stack took this renewal for a repeat of an earlier one, so nothing changed. This is tried again every hour
`http_401`	The stack refused a token during this renewal. The token in use stays, and this is tried again every hour
`key_replicating`	The search head this reached has not received TrackMe’s keys from the others yet. This is tried again every hour

Any other reason reads as this: the token could not be renewed, the token in use keeps working, this is tried again every hour, and the add-on’s log has the reason.

The page reads a status file, $SPLUNK_HOME/var/run/trackme_broker/<input>.json, mode 600, because a forwarder may not run a KV store. While a connection waits to be accepted that file holds the pairing code, which never reaches a log.

What the target test says¶

Test on a target sends nothing but a TLS handshake. It resolves the name once, screens the answer as a call would, and connects to the addresses it screened. It uses the same trust a call would use. It judges the certificate even with verification off, because whether turning it on would work is what you need to know. It needs an administrator on the forwarder. The page shows a sentence for each result. These are the results behind them, as the add-on’s /trackme_broker/test_target endpoint answers.

Result	Meaning
`ok`	Reachable, and the certificate checks out
`reachable_unverified`	Reachable, and verification is off on this target. The page says whether this certificate would pass
`self_signed`, `certificate_untrusted`	Copy the certificate or its issuing CA to this forwarder, and name that file as the CA file
`hostname_mismatch`	Use a name the certificate lists, or have it reissued
`certificate_expired`, `certificate_not_yet_valid`	Renew it at the destination, or check the clocks
`ca_file_unreadable`	The CA file cannot be read, or holds no certificate. It must be a PEM file the Splunk user can read
`name_not_resolved`, `network_unreachable`	This forwarder cannot resolve the name, or has no route to the address
`connection_refused`, `connect_timed_out`	The host refused the connection, or nothing answered within 8 seconds. Check the port and any firewall
`not_tls`, `tls_handshake_failed`, `tls_timed_out`	The port does not speak TLS, TLS could not be negotiated, or the handshake took more than 8 seconds. A plain HTTP port reads as `not_tls`
`target_refused`	An instance metadata or special-purpose address, always refused. It is not probed
`wildcard_host`	A wildcard has no single host to test. Enter a host it covers
`invalid_url`	It needs `https://host`, with an optional port and path

Limits and constants¶

How large a call can be¶

The lowest of these applies to a response. Above any of them, the call fails with response_too_large.

Ceiling	Value
What the stack asks for on a remote Splunk SDK call, such as a remote search	256 MiB. A search’s results come back as one HTTP response
What the stack asks for on every other call: `requests`-based calls, Cribl connections, webhook accounts	64 MiB
The forwarder’s own setting, Largest response (MB) on the connection	256 by default, and at most 256. It can only lower the stack’s limit
The chunks	256 chunks of 768 KiB of gzip: 192 MiB compressed. Search results normally compress well enough for 256 MiB to fit

A direct account has no size limit on that one response, so a broker adds this ceiling. A search that returns more than this through a broker fails, where the same search on a direct account would not.

Keep Largest response (MB) the same on every forwarder of a broker. The lower of the stack’s limit and the forwarder’s wins, so forwarders with different values can give a call different outcomes, depending on which one pulled it. TrackMe compares the forwarders’ target lists, and not this setting, so nothing flags a difference.

A request body stays at 8 MiB and is not chunked: it is one KV row and one pull, and a larger one is refused at once with request_too_large. The KV store’s own limit of 16 MB per record does not change. Responses fit under it because each chunk is its own row.

Every limit¶

What	Value
Queued requests per broker	500
Request body	8 MiB decoded, not chunked
Request headers	128 headers, 64 KiB in all
URL length	8192 characters
`timeout_seconds`, and each of the connect and read timeouts	At most 3600 s. By default 60 s, and 30 s to connect
`response_limit` in the envelope	64 MiB by default, 256 MiB for an SDK call. The stack refuses more than 256 MiB
Largest response (MB), `response_limit_mb`	256 by default, from 1 to 256, in MiB. The add-on applies the lower of this and the envelope’s limit, so this setting can only lower a limit
Concurrent requests, `max_in_flight`	4 by default, from 1 to 64
Hold of a `pull`	The add-on asks for 20 s. The stack holds at most 25 s
Requests per `pull`	At most 20, and 16 MiB of sealed envelopes
Chunks	At most 256. The add-on cuts at 768 KiB of gzip, 1 MiB of base64, so a response that compresses badly stops at 192 MiB of gzip. The stack accepts up to 4 MiB of base64 and a 6 MiB seal
Lease, and `progress`	60 s, renewed every 20 s
Attempts on a request	3
Sweep for ended leases, and heartbeat writes	Every 15 s
Offline after no heartbeat	90 s
Time window of a signed call	300 s
A connection, its targets and whether its input is still enabled are read again	Every 60 s
A change to Largest response (MB) or Concurrent requests	Applies within about three minutes. Running calls get up to 60 s to finish
Retries of a `push` or a `fail`	5, each with a timeout of 120 s. A key still on its way does not count: those go on until the report lands or its claim ends
Retries of the channel	Backs off up to 60 s, and straight to 60 s for a refusal a retry cannot change
A response nobody released	Removed after 1 hour
Rows nobody came back for, failures included	Removed after 1 day, checked at most every 10 minutes
A silent instance	Removed after 30 days, and what TrackMe minted for it is revoked
The forwarder’s token	Renewed right after pairing, then every 7 days by default, from 1 to 365, and with under 30% of a stated life left
A token renewal that failed	Tried again every hour
Broker name	Starts with a letter or a digit, at most 64 characters: letters, digits, dash and underscore

Concurrent requests and Largest response (MB) are on the connection’s edit dialog. Together they are what the forwarder may hold in memory at once. The process is built with them when it starts, so a change restarts it by itself. It stops taking work, goes on renewing its leases, gives what is running up to 60 seconds to finish, and stops. splunkd then starts it again at the next interval. In total, the change applies within about three minutes. A call still running after those 60 seconds is cut, with its normal outcome: a call that is safe to repeat goes back to the queue, and any other fails with broker_lost. The connection’s other fields and the targets apply within a minute, with no restart.

On the dialog, only a limit you changed is validated and saved. If the connection saves and its limits do not, the dialog says so.