ctrl-exec - Installation and Operations

This document is the authoritative reference for all ctrl-exec-dispatcher and ctrl-exec-agent commands. It covers every mode, option, and environment variable for both binaries.

For installation and initial setup, see INSTALL.md. For a hands-on introduction to what the system can do, run the ctrl-exec-demonstrator script on a paired agent - see INSTALL.md for instructions on enabling it.

ctrl-exec-dispatcher

The ctrl-exec-dispatcher binary runs on the dispatcher host. It manages the CA, handles agent pairing, and dispatches commands to paired agents.

Synopsis

ctrl-exec-dispatcher <mode> [options] [args]

Global options

These options apply to all modes.

--config <path>: Path to ctrl-exec.conf. Default: /etc/ctrl-exec-dispatcher/ctrl-exec.conf.
--port <n>: Override the default agent port (7443) for all hosts in this invocation. Individual hosts can also override their port with <host>:<port> syntax.
--json: Output results as JSON. Applies to run, ping, and list-agents.
--username <n>: Username to include in the request context. Defaults to $USER. This is an advisory field - it is forwarded to auth hooks and to the agent unchanged, but is not authenticated or verified by the dispatcher. Its intended use is to carry an identity assertion that the auth hook can forward to an external authentication service alongside the token, allowing that service to verify whether the claimed identity is consistent with the token's authority. Do not use username alone as an access control basis. See SECURITY-OPERATIONS.md for the recommended pattern.
--token <token>: Auth token to include in the request context. Defaults to the $ENVEXEC_TOKEN environment variable if set. Consumed by auth hooks on the agent side.

Environment variables

ENVEXEC_TOKEN: Default auth token. Equivalent to --token. Useful for scripted or automated invocations where passing a token on the command line is undesirable.
USER: Default username. Overridden by --username.

run

Run an allowlisted script on one or more agents in parallel.

ced run <host>[:<port>] [<host>...] <script> [-- <arg>...]

The script name must match an entry in the agent's scripts.conf allowlist. Everything after -- is passed to the script as positional arguments. Arguments are evaluated on the dispatcher host before being sent - use -- to pass static strings, not shell expressions intended to run on the agent.

The dispatcher sends a JSON context object to the script via stdin on the agent. This includes the script name, username, token, arguments, timestamp, peer IP, and request ID. See INSTALL.md for the full context structure.

# Single host
ced run web-01 deploy-app

# Multiple hosts in parallel
ced run web-01 web-02 web-03 deploy-app

# Pass arguments to the script
ced run db-01 backup-mysql -- --database myapp

# Custom port for one host
ced run web-01:7450 deploy-app

# Identify the operator explicitly
ced run web-01 deploy-app --username alice

# Pass an auth token
ced run web-01 deploy-app --token mytoken
ENVEXEC_TOKEN=mytoken ced run web-01 deploy-app

# JSON output
ced run web-01 deploy-app --json

Output shows per-host status, exit code, round-trip time, stdout, and stderr. Exit code is 0 if all hosts succeeded, 1 if any host failed or returned a non-zero exit.

Asynchronous runs (`--async`)

For jobs that run longer than the dispatcher read_timeout (default 60s), submit with --async:

ced run db-01 db-02 long-reindex --async

The agent starts the script detached and the command returns immediately with a request id instead of waiting for output:

Submitted async run  req:a1b2c3d4e5f60718
==> db-01  [accepted  12ms]
==> db-02  [accepted  10ms]
Check progress with: ced status a1b2c3d4e5f60718

Each host reports accepted, BUSY (the agent is already running that script), or ERROR. Exit code is 0 if any host accepted, 1 if none did. Retrieve results later with status or wait (below). Because the job is detached (setsid + the agent unit's KillMode=process), it survives the connection close and an agent restart; its result is held on the agent and fetched on demand. Agent-side concurrency still applies: a second async run of the same script on a host is refused while the first is in flight.

Dispatch host resolution

When a host argument matches a registered agent name, both the connect address and the port are resolved from the registry (set at pairing).

Address, from the agent's lookup_by field (see approve --lookup-by):

hostname (default): connect to the agent's registered hostname.
ip: connect to the IP recorded at pairing. Use this when the agent-reported hostname does not resolve from the dispatcher.

Port, from the agent's port field (see approve --agent-port): the stored operational port is used automatically, so an agent on a non-default port is reachable without passing --port each time.

A host argument that is not a registered agent, or one given with an explicit host:port, is used verbatim (DNS resolution and the explicit port as before) — so ad-hoc and one-off-port targets still work. An explicit host:port overrides the stored port; the global --port applies only to hosts whose port is not otherwise determined. mTLS is unaffected: the agent certificate is validated against the CA, not matched to the connect address.

The registry name is canonical: run, ping, and discovery display and key results by the name the agent was addressed by (its registry name), regardless of the address actually connected to. When the agent reports a different hostname for itself, that reported name is shown in brackets — for example web-01 (vm-7a3f) — or carried as reported_hostname in JSON / discovery output.

status

Show the current state of an asynchronous run submitted with run --async.

ced status <reqid>

On each call the dispatcher fetches results from every host still running, merges them into the stored record, and prints the per-host state:

Request a1b2c3d4e5f60718  (script: long-reindex)  [PENDING]
==> db-01  [done  exit:0]
reindex complete
==> db-02  [running]

Per-host states: done (with exit code, stdout, stderr), running / accepted (still in progress), EXPIRED (the agent purged its result before it was fetched — results live 24h), BUSY or ERROR (the job never ran there). The header shows COMPLETE once no host is still running. --json emits the full record. Exit code is 0 when the record is found, 1 if the reqid is unknown.

wait

Poll an asynchronous run until it completes, then print the final result.

ced wait <reqid> [--timeout <seconds>]

wait repeatedly aggregates status (every 2s) until every host has finished or the timeout elapses (default 300s). Output is the same as status. Exit code:

0 — complete, all hosts succeeded (exit 0)
1 — complete, but some host failed (non-zero exit, busy, error, or expired)
2 — timed out while still pending

This makes wait suitable for scripting a submit-then-block flow:

reqid=$(ced run db-01 db-02 long-reindex --async --json | jq -r .reqid)
ced wait "$reqid" --timeout 600

ping

Test mTLS connectivity to one or more agents and report cert expiry and agent version.

ced ping <host>[:<port>] [<host>...]

# Single host
ced ping web-01

# Multiple hosts in parallel
ced ping web-01 web-02 web-03

# JSON output
ced ping web-01 --json

Output columns: host, status (ok/error), round-trip time, cert expiry, agent version.

pairing-mode

Start the pairing listener on port 7444. Agents call request-pairing to submit a CSR; this mode receives the request, displays it for operator approval, and signs and returns the certificate on approval.

ced pairing-mode

The prompt shows the requesting agent's hostname, IP, request ID, and a 6-digit confirmation code with its version, e.g. 482917 (v2). The agent displays its code once connected. Verify both match before approving. A v2 code is bound to the certificate the agent saw on the pairing connection, so a relay in between shows a different code on each side. An agent older than 0.13 gets the older v1 code, labelled (v1, pre-0.13 agent), which a relay can reproduce; see SECURITY.md, Pairing Security. Enter a to approve, d to deny, or s to skip and leave the request pending. Press Ctrl-C to stop pairing mode.

Pairing mode processes one request at a time interactively. For unattended approval workflows, use list-requests and approve.

Verified pairing. On start, pairing mode prints this dispatcher's CA fingerprint and the agent command for verified pairing, ready to paste:

Pairing mode active on port 7444 (auto-stops in 600s). Ctrl-C or 'quit' to stop.
Dispatcher CA fingerprint: SHA256:3f2a...c91e
On the agent (verified pairing; use it on any untrusted network):
    ctrl-exec-agent request-pairing --dispatcher d1.example.com --dispatcher-fingerprint SHA256:3f2a...c91e
Waiting for pairing requests...

The host in the command is this host's name; replace it with the name or address the agent reaches the dispatcher by if they differ. --port is added when the pairing port is not 7444. The listener sends its CA certificate after its own, which is what lets the agent check it. If the CA cannot be used (no ca.crt in the config directory, or it did not issue the certificate named by cert), the banner says Verified pairing unavailable: <reason>., logs ACTION=pairing-ca-unavailable, and pairing works as before without it. See ca-fingerprint and request-pairing --dispatcher-fingerprint.

Automatic timeout. Pairing mode always auto-stops after an absolute timeout so the pairing port cannot be left open indefinitely. The default is 600 seconds; lower it with --timeout <seconds> (1–600) or the pairing_timeout config key. The window can be shortened but never extended beyond 600 seconds.

ced pairing-mode --timeout 120   # auto-stops after 2 minutes

Background start/stop. Run the listener detached instead of in the foreground:

ced pairing-mode start           # detach; auto-stops at the timeout
ced pairing-mode stop            # stop a backgrounded session, close the port

start reports the PID and the stop deadline, followed by the same verified pairing lines, then returns. Only one background session runs at a time (tracked by a pidfile under /var/lib/ctrl-exec). stop is a no-op, with a clear message, if nothing is running.

list-requests

List pending pairing requests that have not yet been approved or denied.

ced list-requests

Output columns: request ID, hostname, source IP, reverse DNS, confirmation code with its version (v2, or v1, pre-0.13 agent). Verify the confirmation code matches the 6-digit code displayed on the agent before approving. approve repeats the code and its version.

A request from an agent already paired with another dispatcher (see Pairing with another dispatcher under request-pairing) is followed by two lines: Keeps cert: with the SHA-256 fingerprint of the agent's certificate, and Issued by: with the fingerprint and subject of the CA that issued it. Compare them with the agent's own display before approving.

approve

Approve a pending pairing request by ID. Signs the agent's CSR and delivers the certificate on the agent's next poll.

For an agent that keeps its certificate (already paired with another dispatcher), nothing is signed: the CA that issued the agent's certificate is pinned in the agent's registry entry (agent_ca, agent_ca_fpr), this dispatcher verifies that agent's TLS certificate against that CA alone, and it never renews the certificate (the issuing dispatcher does; this dispatcher records the expiry the agent reports on ping). approve says so. When the issuing CA is this dispatcher's own (an agent re-pairing with the dispatcher that issued its certificate) nothing is pinned and the agent is an ordinary agent of this dispatcher.

ced approve <reqid>
ced approve <reqid> --lookup-by ip
ced approve <reqid> --ip 192.168.1.10
ced approve <reqid> --agent-port 7450

The request ID is shown by list-requests and pairing-mode.

The IP stored for the agent is the address the agent reports for itself at pairing time (the source address of its connection to the dispatcher). The dispatcher falls back to the connection's own source IP only if the agent does not report one — note that source IP is unreliable when a NAT sits in front of the dispatcher (for example the dispatcher running in a Docker container, where every agent's connection appears to come from the bridge gateway). Use --ip to set the address explicitly in that case.

--lookup-by <ip|hostname>: How dispatch resolves this agent's connect address (stored in the registry as lookup_by). hostname (default) connects to the agent's registered hostname; ip connects to the IP recorded at pairing. Use ip when the agent-reported hostname does not resolve from the dispatcher. This overrides any value the agent suggested with request-pairing --lookup-by. See Dispatch host resolution under run.
--ip <addr>: The address to record for this agent (registry ip), used by dispatch when lookup_by is ip. Overrides the agent's self-reported source IP and the connection source IP. Set this when a NAT in front of the dispatcher hides the agent's real address. Accepts IPv4 or IPv6.
--agent-port <n>: The agent's operational port, stored in the registry as port and used by dispatch so the agent is reachable without a per-command --port. Overrides the port the agent reported at pairing (its configured serve port). Useful when the dispatcher reaches the agent on a different port than the agent binds — for example a port-mapped container. Default 7443.

deny

Deny a pending pairing request by ID. Removes the request without signing.

ced deny <reqid>

ca-fingerprint

Print the SHA-256 fingerprint of this dispatcher's CA certificate (ca.crt in the config directory), for verified pairing.

ced ca-fingerprint
# SHA256:3f2a...c91e

One line on stdout, SHA256: then 64 lowercase hex digits, as request-pairing --dispatcher-fingerprint takes it. It is the fingerprint of the CA, not of the dispatcher certificate, so it does not change on rotate-cert. Fails, with the reason on stderr, if there is no CA certificate or it did not issue the dispatcher's certificate (cert in ctrl-exec.conf).

list-agents

List all registered (paired) agents.

ced list-agents
ced list-agents --json
ced list-agents --tags env=production
ced list-agents --tags env=production,role=database

Output columns: hostname, IP address, operational port, lookup mode, paired timestamp, cert expiry.

--tags <key=value[,key=value...]>: Filter to agents whose cached tags match every given key=value pair (AND). Tags are read from the registry cache, not by querying agents live, so the filter is fast and works offline. The cache is populated and refreshed by discovery (which reads each agent's live tags from /capabilities); a newly paired agent shows no tags until the first discovery against it. Tag values are matched exactly; this filter syntax cannot express commas within a value. Tags appear in the --json output under the tags field.

With --json, returns an array of objects with fields:

hostname: Agent hostname as registered at pairing time.
ip: IP address of the agent at last contact.
port: The agent's operational port (default 7443). Used by dispatch.
lookup_by: How dispatch resolves this agent: hostname (default) or ip. See Dispatch host resolution under run.
tags: Object of the agent's key/value tags, cached from /capabilities on the last discovery. Empty {} until the first discovery against the agent. Used by --tags.
paired_at: ISO 8601 timestamp of when the agent was paired.
cert_expiry: ISO 8601 timestamp of the agent certificate's notAfter date.
serial: Truncated hex serial of the agent's current certificate.

edit-agent

Edit a registered agent's dispatch fields, or rename it, without re-pairing. Only the dispatch-relevant fields are editable; the certificate and serial tracking are left intact. At least one change flag is required.

ced edit-agent <name> --ip 10.0.0.42
ced edit-agent <name> --agent-port 7450 --lookup-by ip
ced edit-agent old-name --rename new-name

--rename <new-name>: Move the registry entry to a new canonical name. Fails if the new name is already in use or is not a valid name (letters, digits, ., -, _; must start with a letter or digit).
--ip <addr>: Update the stored address used when lookup_by = ip.
--agent-port <n>: Update the stored operational port (1–65535).
--lookup-by <ip|hostname>: Update how dispatch resolves this agent. See Dispatch host resolution under run.

--json prints the updated record. The agent's certificate is unaffected; renaming or re-addressing does not require the agent to re-pair, since mTLS validates the certificate against the CA rather than the connect address.

setup-ca

One-time initialisation of the dispatcher CA. Generates the CA key and self-signed certificate used to sign all agent certificates. Run once on the dispatcher host before any pairing.

ced setup-ca

Writes to /etc/ctrl-exec-dispatcher/. Does not overwrite an existing CA. The CA's subject is CN=ctrl-exec CA <dispatcher-id> <8 random hex digits> (0.14; earlier releases named every CA ctrl-exec CA), so the CAs of several dispatchers never share a name in an agent's CA bundle. Every certificate the CA issues carries Subject and Authority Key Identifiers, so a verifier holding several CAs finds the issuer by key. An existing CA is not regenerated.

From 0.14 every certificate the CA issues is also marked with its role: dispatcher for the dispatcher's own (setup-ctrl-exec, rotate-cert, automatic rotation), agent for every certificate signed from a CSR (pairing approval, renewal), whatever the CSR asks for. Verified pairing refuses an agent-marked certificate on the pairing port. Certificates issued earlier stay unmarked, and valid, until the agent's next renewal or the dispatcher's next rotation; nothing is reissued on upgrade. See SECURITY.md, Certificate Roles.

setup-ctrl-exec

Generate the dispatcher's own key and certificate, signed by the CA. Run after setup-ca. If an existing cert is found and registered agents exist, the command displays the agent count and requires confirmation before proceeding - replacing the cert changes its serial, and setup-ctrl-exec does not tell the agents: each refuses the new certificate until it is paired again. To replace the certificate of a dispatcher with paired agents, use rotate-cert, which does tell them.

ced setup-ctrl-exec

Writes to /etc/ctrl-exec-dispatcher/.

rotate-cert

Rotate the dispatcher certificate immediately, in three steps, and report per-agent results:

Stage: generate the new key and certificate as <key>.next and <cert>.next beside the cert and key named in ctrl-exec.conf, copy the current pair to <cert>.prev and <key>.prev, record both serials in /var/lib/ctrl-exec/rotation.json, and mark every registered agent pending. Nothing served has changed yet.
Broadcast: tell every pending agent, in parallel and at its registered address and port, to add the new serial to its trusted-dispatcher map. The broadcast is authenticated with the current certificate - an agent accepts a serial only from a dispatcher certificate it already trusts.
Switch: install the .next pair as the cert and key. Every later connection to an agent uses the new certificate; long-running processes (ctrl-exec-api) need no restart, as each connection loads the files when it is made.

ced rotate-cert

Before 0.14 the new certificate replaced the old one before the broadcast, so the broadcast presented a serial no agent trusted: every agent refused it (serial-broadcast-fail ... serial mismatch) and then refused the dispatcher. Such an agent needs pairing again; doctor names it (agent-missed-rotation).

A rotation that was interrupted (the process killed between steps) is finished, not replaced, by the next rotate-cert or maintain run (or the pairing listener's check loop): the broadcast is sent again (adding a serial an agent already has changes nothing) and whichever .next file is left is installed. doctor reports it until then (rotation-unfinished).

The new certificate is marked with the dispatcher role (0.14; see setup-ca), so rotating a certificate issued before 0.14 is what marks it. It also carries an Authority Key Identifier, which a certificate issued before 0.14 on an OpenSSL 1.1 host lacks; upgrading keeps the old certificate until the next rotation. An agent that trusts two CAs with one name needs it (see Two CAs with one name under request-pairing).

Agents that were unreachable during the broadcast stay pending and refuse the new certificate until they are reached. Each maintain run (and the pairing listener's check loop) retries them, authenticated with the previous certificate they still trust. Agents that remain unreachable after cert_overlap_days are marked stale and require re-pairing. Once the overlap window has passed, maintain removes the old serial from the agents that took the new one and deletes the .prev pair. Running rotate-cert again while agents of the last rotation are still pending keeps the previous pair they trust, so they are still reached, and retires the serial in between with the old one.

The broadcast uses the agent's built-in /rotate-serial operation, so it needs no scripts.conf allowlist entry — cert rotation is a first-class control-plane operation, not an allowlisted script.

maintain

Run one maintenance pass: ping every registered agent (which triggers certificate renewal for agents past half-life), rotate the dispatcher's own cert if it nears expiry, broadcast the serial to agents that have not taken it (finishing an interrupted rotation), expire agents long out of contact, retire the previous serial once the overlap window has passed, and purge expired async run records. This is what ctrl-exec-maintenance.timer runs; it needs root for CA-key access.

ced maintain [--json]

Each phase (ping, rotate, broadcast, expire, retire, purge) runs independently: a failure is logged at ERR (ACTION=maintain PHASE=<phase>) and the remaining phases still run.

Output on success:

Maintenance run: 3/3 agents pinged, dispatcher cert checked.

If any phase failed, a second line names them and the exit code is 1:

Maintenance FAILED in phase(s): rotate, purge

--json emits {"ok":1,"agents":3,"pinged":3,"failed":[]}. When a phase failed, ok is 0 and failed lists the phase names in run order. The failed key is always present.

Exit code is 0 when every phase succeeded, 1 if any phase failed.

serial-status

Show the current dispatcher serial, previous serial, rotation timestamps, overlap expiry, and per-agent serial state.

ced serial-status
ced serial-status --json

Output columns: hostname, status (current/pending/stale/unknown), last confirmed timestamp, serial (truncated).

Status values:

current - agent has confirmed the current serial
pending - broadcast attempted but not yet confirmed (agent may be offline)
stale - overlap window expired without confirmation; re-pair required
unknown - agent has never received a serial update

doctor

List every problem this dispatcher's files show, each with the command that fixes it. Use it after an upgrade, or when something does not work and the symptom does not say why.

ced doctor [--fix] [--json] [--brief]

One line per finding, problems first, then a summary line:

PROBLEM  The auth hook /etc/ctrl-exec-dispatcher/auth-hook allows every request (it is the shipped example, or does nothing but exit 0): every API and CLI request is authorised. Fix: add real rules (examples: /usr/share/ctrl-exec/examples/auth-hook.example): sudoedit /etc/ctrl-exec-dispatcher/auth-hook; or comment out auth_hook in /etc/ctrl-exec-dispatcher/ctrl-exec.conf (then api_auth_default applies)
note     The dispatcher config is still at /etc/ctrl-exec and is used there; it works as it is, but has not been moved to /etc/ctrl-exec-dispatcher. Fix: sudo dpkg-reconfigure ctrl-exec-dispatcher (moves it and leaves a symlink at /etc/ctrl-exec)
ctrl-exec-dispatcher doctor: 1 problem, 1 note.

PROBLEM is something that does not work as configured, or is unsafe; note is something that works but is worth knowing, or that could not be checked. Checks (the check id in --json in brackets):

Check	Finding
Config directory	`ced` and `ctrl-exec-api` refuse to start: `/etc/ctrl-exec` beside `/etc/ctrl-exec-dispatcher` (problem, `config-dir`); a config still at `/etc/ctrl-exec`, or a mount there, used in place (note, `config-dir-in-place`, with the command that moves it). `doctor` runs in both cases; the other modes refuse the first
`ctrl-exec.conf`	cannot be loaded (problem, `config`)
CA and certificate	a fresh install with neither (note, `not-set-up`); `ca.crt` or the dispatcher certificate missing (problem, `ca-cert`, `dispatcher-cert`); the certificate expired or due and not rotated, within 30 days or `cert_renewal_days` (problem, `dispatcher-cert-expiry`); the certificate has no Authority Key Identifier (issued before 0.14 on an OpenSSL 1.1 host), so an agent that also trusts another CA with this CA's name refuses it - fix `rotate-cert` (note, `dispatcher-cert-no-aki`; see Two CAs with one name under `request-pairing`)
Auth hook	`auth_hook` names a missing or non-executable file, so every request is denied (problem, `auth-hook-missing`, `auth-hook-not-executable`); the hook allows everything - after comments, nothing but `exit 0`, as the shipped example (problem, `auth-hook-allow-all`)
API listener	settings `ctrl-exec-api` refuses (problem, `api-config`, with its reason); `api_socket` absent (note, `api-socket-absent`: the API is not running); the socket wider than 0660, or not in `api_socket_group` (problem, `api-socket-mode`, `api-socket-group`)
Registry	an agent paired keeping a certificate another dispatcher's CA issued (`approve` pinned the CA), expiring within 30 days: this dispatcher never renews it (problem, `agent-cert-pinned-expiry`); any other agent whose certificate has expired (problem, `agent-cert-expired`)
Certificate rotation	a rotation interrupted before its new certificate was installed (problem, `rotation-unfinished`; `maintain` or `rotate-cert` finishes it); an agent that missed a rotation and refuses this dispatcher: `stale` (the overlap window closed), or `pending` with no previous certificate kept to reach it with, as a rotation before 0.14 left it (problem, `agent-missed-rotation`; pair it again). An agent `pending` within the overlap window is retried automatically and not reported

--fix also makes the changes that cannot lose data or widen access, and lists each as fixed: the API socket's mode back to 0660 and its group back to api_socket_group, which is what ctrl-exec-api sets at every start. Nothing else is changed (moving the config directory is the installers' job). A second --fix changes nothing.

--json prints one object on stdout and nothing else:

{"findings":[{"check":"auth-hook-allow-all","fix":"...","fixed":false,"severity":"problem","what":"..."}],"fixed":0,"notes":0,"ok":false,"problems":1,"program":"ctrl-exec-dispatcher"}

--brief lists problems and fixes only, and counts the notes; with nothing to fix it prints one line. The package upgrade and install.sh run doctor --brief after an upgrade and print the result; they never use --fix, and its result never fails the install.

Each finding is also logged: ACTION=doctor-finding PROGRAM=ctrl-exec-dispatcher SEVERITY=<problem|note> CHECK=<id> WHAT=<text> FIX=<command> (FIXED=1 when --fix made it), at WARNING for a problem, INFO otherwise.

Without root it checks what it can read (members of the ctrl-exec group read the config and the registry) and reports the rest as a note: Could not read <path> (permission denied).

Exit code: 0 when no problem was found (notes may remain), 1 when one was, 2 on bad usage.

ctrl-exec.conf

Configuration file for the dispatcher and ctrl-exec-api processes. Default path: /etc/ctrl-exec-dispatcher/ctrl-exec.conf.

Key settings:

cert, key, ca: Paths to the dispatcher's TLS certificate, private key, and CA certificate. Required for all mTLS operations.
dispatcher_id: Stable identity for this dispatcher. Defaults to the dispatcher hostname. Delivered to each agent at pairing and recorded against the dispatcher's cert serial in the agent's trusted-dispatcher map. Agent permission and attribution key on this id, so it must remain stable across cert rotations - set it explicitly if the hostname may change.

dispatcher_id = ctrl-exec-primary

read_timeout: How long (in seconds) the dispatcher waits for a response from an agent before reporting a timeout error. Default: 60. The script continues running on the agent after a timeout - only the dispatcher's ability to receive the output is affected. Raise this value for scripts that are expected to take longer than 60 seconds.

read_timeout = 120

timeout: Deprecated alias for read_timeout. Accepted for backward compatibility.
max_parallel: Maximum concurrent agent connections per fan-out operation (run/ping/discovery over a fleet). Default: 64. Bounds how many TLS clients are forked at once so a large fleet does not exhaust file descriptors; hosts beyond the cap queue. Read by Exec::Engine.

max_parallel = 64

max_input_bytes: Largest total size, in bytes, of the values in a run's confidential input (see POST /run in API.md). Default: 1048576 (1 MiB). A larger input is refused with a 400 at the API. The API's request body limit is 1 MiB plus this value. The agent has its own max_input_bytes and refuses what exceeds it; set them alike. At most 32 fields are allowed, regardless of this setting.

max_input_bytes = 1048576

api_socket: Absolute path of a unix socket for same-host API callers. Optional; not set by default. The socket is created mode 0660, so file permissions decide who may call: the socket's owner (the API's user) and members of api_socket_group. Requests on the socket are handled exactly as on TCP and pass the same auth hook, which sees source_ip as unix:<path> and the caller's uid (from the kernel, SO_PEERCRED) as peer_uid / ENVEXEC_PEER_UID. A run started over the socket stores that uid with its submitter; on GET /status/{reqid} the hook also gets it as owner_peer_uid / ENVEXEC_OWNER_PEER_UID (null / empty for a run started over TCP or recorded before this), beside the current caller's peer_uid, so a hook can allow "same uid as the starter". It is for hooks only and is not in the status response. The packaged systemd unit creates /run/ctrl-exec-api for it. At start-up an existing socket nothing answers on (left by a crash) is replaced; a socket another process answers on, or a path that is not a socket (a file, directory or symlink), stops start-up and is left alone. The file is removed when the API stops. Confidential run input (input on POST /run) should travel only over this socket or TLS.

api_socket = /run/ctrl-exec-api/api.sock

api_socket_group: Group of the api_socket file, by name or number. Default: the API's own group (ctrl-exec under the packaged unit). The API's user must be a member of the group. Setting it without api_socket is refused.
api_port: Port for the ctrl-exec-api TCP listener. Default: 7445. off turns the TCP listener off, leaving only api_socket (which must then be set). The socket and TCP listeners can run together.
api_tls_cert, api_tls_key: TLS certificate and key, provided by the operator, for the TCP listener. Optional: with neither set the TCP listener is plain HTTP. With both set it is HTTPS only. Setting one without the other, naming a file the API's user cannot read, a certificate and key that do not match, or setting them with api_port = off stops start-up with an error (exit 78) rather than falling back to plain HTTP. Use a certificate whose subject alternative name matches the host name clients connect to, from a CA the clients trust; clients verify it (ctrl-exec-cli --ca FILE, curl --cacert FILE). The names api_cert and api_key are refused at start-up (exit 78) with a message naming the new key.
api_bind: Network address the API server binds to. Default: 127.0.0.1 (localhost only). Set to 0.0.0.0 to accept connections on all interfaces, or to a specific interface address. Only change this if external clients need direct access; prefer a reverse proxy for internet-facing deployments.

api_bind = 0.0.0.0

api_auth_default: Controls API behaviour when no auth_hook is configured. Accepts deny (default) or allow. With deny, all requests are rejected until a hook is configured. With allow, all requests are authorised without a hook - suitable only for isolated networks. When a hook is configured this setting has no effect.

api_auth_default = allow

auth_deny_generic: When enabled (1/yes; default off), the API returns a generic 403 with {"error":"forbidden"} for denied run, ping, and discovery requests, withholding the specific denial reason and code from the caller to reduce information disclosure. The full reason is still logged server-side. Off returns the detailed reason and code.

auth_deny_generic = 1

Cert rotation settings

cert_days: Lifetime of the dispatcher certificate in days. Default: 365. Applied when generating a new cert via setup-ctrl-exec or automatic rotation.
cert_renewal_days: Begin renewal this many days before the dispatcher cert expires. Default: 90. With cert_days = 365, renewal begins at day 275 of the cert's life.
cert_overlap_days: After rotation, continue retrying agents that missed the serial broadcast for this many days before marking them stale (requiring re-pair); the old serial stays trusted, and the previous certificate kept, for as long. Default: 30. Configurable to accommodate fleets with agents that are offline for extended periods.

cert_overlap_days = 60

cert_check_interval: Seconds between internal cert expiry checks. Default: 14400 (4 hours). Reduce for testing. The check is lightweight: read the cert, compare notAfter to now.
auth_hook: Path to an executable called before every run and ping operation. Receives request context as JSON on stdin. See INSTALL.md for the full interface contract.

agent.conf

Configuration file for the ctrl-exec-agent process. Default path: /etc/ctrl-exec-agent/agent.conf.

Key settings:

port: Port the agent listens on for mTLS connections from the dispatcher. Default: 7443.
cert, key, ca: Paths to the agent's TLS certificate, private key, and CA certificate. Like port, these are read once at start: a change needs a restart (SIGHUP keeps the running values and logs that a restart is needed).

The ca file is the agent's CA bundle: one PEM certificate for each dispatcher CA the agent trusts, in the order they were paired. Pairing appends the dispatcher's CA certificate to /etc/ctrl-exec-agent/ca.crt unless a certificate with the same SHA-256 fingerprint is already there; it never replaces the CAs of dispatchers paired before. An agent paired before 0.14 holds one certificate, a bundle of one, and needs no change. The TLS listener accepts a peer certificate issued by any CA in the bundle; which dispatcher the peer is, if any, is decided by the trusted-dispatcher map below, by issuing CA and serial. A renewed agent certificate must come from the CA that issued the current one. The running agent watches the file: when it changes (a pairing appends to it) the listener is given the new set of CAs at the next connection, with no restart (ACTION=ca-bundle-reload). A bundle with no readable certificate leaves the listener with the CAs it had.

revoked_serials: Path to a file listing revoked TLS certificate serial numbers, one per line. Connections presenting a revoked cert are rejected immediately after the mTLS handshake, before any request is processed. Reloaded on SIGHUP without restart.

A line is either <serial>, which revokes that serial whatever CA issued the certificate (the only form before 0.14, still honoured), or <ca-fingerprint> <serial>, which revokes it only in certificates issued by that CA, so the same serial from another dispatcher's CA still runs. <ca-fingerprint> is the CA certificate's SHA-256 fingerprint as 64 hex characters (colons removed):

openssl x509 -in ca.crt -noout -fingerprint -sha256 | cut -d= -f2 | tr -d :

Accepted serial formats (all normalised to lowercase hex on load):

Plain hex: deadbeef
Colon-separated: DE:AD:BE:EF (as returned by some tools and IO::Socket::SSL)
0x-prefixed: 0xdeadbeef
serial=-prefixed: serial=DEADBEEF (as output by openssl x509 -serial)
Decimal integer: 3735928559

Serials the CA issues (since 0.13.0) are random 32-digit lowercase hex values with no leading zero; certificates issued earlier keep their (short, sequential) serials, and both forms match through the same normalisation.

Lines beginning with # are treated as comments.

Default: /etc/ctrl-exec-agent/revoked-serials. A missing or empty file means no certs are revoked.

revoked_serials = /etc/ctrl-exec-agent/revoked-serials

trusted_dispatchers_path: Path to the trusted-dispatcher map. Each line maps a dispatcher certificate, named by its issuing CA and its serial, to a dispatcher id: <ca-fingerprint> <hex-serial> <dispatcher-id>, where <ca-fingerprint> is the SHA-256 fingerprint (64 hex characters) of the CA in the ca bundle that issued the dispatcher's certificate. Written automatically by request-pairing and updated in place during cert rotation - do not edit manually. Pairing appends an entry, so an agent can serve multiple dispatchers natively; re-pair to add another. Re-pairing a dispatcher id under a different CA ends that id's entries under its old CA. The /run, /ping, /result, /rotate-serial and /capabilities endpoints reject a peer unless both its issuing CA and its serial match an entry, so a certificate from another trusted CA that carries a trusted serial is refused; the matched dispatcher identity drives permission and attribution. A dispatcher cert rotation updates the map automatically over the run channel (the rotated serial is added under the caller's CA, then the old serial removed after the overlap window), so no re-pairing is needed for a reachable agent.

Upgrade: before 0.14 a line was <hex-serial> <dispatcher-id>. The agent reads that form and, on first load (start or reload), rewrites each such line with the fingerprint of the CA in its ca file and logs ACTION=trusted-map-upgrade. No operator step. If the map cannot be rewritten the upgrade holds in memory, a warning is logged and it is retried on the next load.

Default: /var/lib/ctrl-exec-agent/ctrl-exec-dispatchers. The map lives in the agent-writable state directory, not the root-owned config dir, so the non-root agent service user can rewrite it in place during cert rotation. Reloaded on SIGHUP.

dispatcher_cn: Removed. Previously used to restrict /capabilities by cert CN. Replaced by the trusted-dispatcher map via trusted_dispatchers_path.
script_dirs: Colon-separated list of absolute directory paths. If set, any script in scripts.conf whose path does not fall under one of these directories is rejected at load time and re-validated at execution time. A run of a script rejected this way answers script not permitted: '<path>' is not under script_dirs (<dirs>), and ctrl-exec-agent self-check lists it as SKIPPED (see "script not permitted" in TROUBLESHOOTING.md).

script_dirs = /opt/ctrl-exec-scripts:/usr/local/lib/ctrl-exec-scripts

script_ownership: warn (the default) or strict; any other value is a configuration error (the agent and the executor both refuse to start). What happens to a script that breaks the ownership rule - the script and every directory from / down to it owned by root and not group- or other-writable (SECURITY.md, "Script ownership").

warn: the script stays allowlisted and runs. The reason is logged once each time scripts.conf is loaded (start and reload) as ACTION=script-ownership REASON=... SCRIPT=... at WARNING, and ctrl-exec-agent self-check shows it next to the script ([warning: '/srv/scripts' is group-writable]). Nothing is added to a run's output.
strict: the script is refused. The agent skips it at load (script not permitted: <reason>, listed by self-check as SKIPPED) and checks again before each run; the executor refuses it with exit 126.

warn accepts that a user who can write the script or a directory on its path can change what a profile runs. Set strict once self-check shows no warnings.

script_ownership = strict

auth_hook: Path to an executable called before every run request on the agent, after allowlist validation. Enables independent downstream token validation separate from the dispatcher's own hook.

The hook receives request context as a JSON object on stdin and as individual environment variables. Exit codes:

0 - authorised, request proceeds
1 - denied (generic refusal)
2 - bad credentials (token not recognised or expired)
3 - insufficient privilege (credentials valid but not permitted for this script)

If no hook is configured, the request is authorised unconditionally - the agent relies on mTLS for identity; the hook is for additional policy enforcement only.

See the Auth hook (agent-side) section under ctrl-exec-agent for context fields, environment variables, and differences from the dispatcher-side hook.

pairing_port: Port the agent listens on during pairing. Default: 7444. Must match the --port value passed to ctrl-exec-agent request-pairing and the port used by ced pairing-mode.

pairing_port = 7444

pairing_max_queue: Maximum number of pairing requests held in the queue at one time. When the limit is reached, incoming requests are rejected immediately with ACTION=pair-reject REASON=queue-full. Default: 10. Raise this value only on deployments where many agents are paired concurrently; the queue is normally short-lived as each request is approved or denied within seconds.

pairing_max_queue = 10

allowed_ips: Comma-separated list of IP addresses or CIDR prefixes (/8, /16, /24) permitted to connect to the agent's mTLS port. Connections from addresses not on the list are dropped immediately after the TCP accept, before the TLS handshake. If unset, all source addresses are accepted.

allowed_ips = 192.168.1.0/24, 10.0.0.1

Invalid entries are logged as ACTION=config-warn at load time and silently skipped.

Rate limiting settings

The agent applies two independent rate limits per source IP. Both are tracked in memory and reset on agent restart or SIGHUP.

Trusted dispatchers are exempt from the volume threshold (since 0.12.4). The accept loop reads the peer certificate serial before the rate check; a serial present in the trusted-dispatcher map (trusted_dispatchers_path, written at pairing) identifies a dispatcher whose every request will be served anyway, and its connections are not counted. Every run, ping, result poll and discovery is a separate connection, so without the exemption ~10 operations in a minute from the one paired dispatcher tripped a 5-minute block. The volume threshold therefore governs CA-signed-but-unpaired peers; the probe threshold, max_children and allowed_ips apply to everyone.

The agent accepts the TCP connection, applies allowed_ips, and only then runs the TLS handshake on the accepted socket, so a failed handshake is counted under the connecting peer's own address. The handshake has a 5-second limit; a client that connects and stays silent is dropped at that point and counted as a failure. The accept loop is single-threaded, so one silent client can delay other new connections by up to those 5 seconds.

A blocked peer receives HTTP/1.0 429 Too Many Requests with a JSON body {"status":"error","error":"rate limited","retry_after":<seconds>} before the connection is closed (earlier releases closed it silently, which the dispatcher reported as no response from child). The agent half-closes after the reply and discards what the client already sent (at most 64 KiB and 1 second in all) before closing, so the reply is not lost to a TCP reset.

The volume threshold blocks IPs that open an unusually high number of connections in a short window — consistent with port scanning or connection flooding. The probe threshold blocks IPs that repeatedly fail TLS handshakes — consistent with certificate probing or brute-force attempts.

rate_limit_volume: Volume threshold in the format limit/window/block. Blocks a source IP for block seconds when it opens more than limit connections within window seconds. Default: 10/60/300 (10 connections in 60 seconds triggers a 5-minute block).

rate_limit_volume = 10/60/300

rate_limit_probe: Probe threshold in the format limit/window/block. Blocks a source IP for block seconds when it produces more than limit TLS handshake failures within window seconds. Default: 3/600/3600 (3 failures in 10 minutes triggers a 1-hour block).

rate_limit_probe = 3/600/3600

rate_limit_disable: Set to 1 to disable all rate limiting. Intended for use during the integration test suite, which opens many connections rapidly from localhost. Do not set in production. (This is the key name Config.pm reads; disable_rate_limit is not recognised.) Since 0.12.4 the paired dispatcher is exempt from the volume threshold, so the suite's own dispatcher no longer needs this unless it also exercises unpaired peers.

rate_limit_disable = 1

Agent tags

Arbitrary key/value metadata attached to the agent and returned in the /capabilities response. Used by the API and any tooling that queries agent capabilities to filter or label agents by environment, role, or location without requiring separate inventory.

Defined in a [tags] section in agent.conf:

[tags]
env = production
role = database
location = ams1

Tags are returned as a JSON object in the tags field of the /capabilities response:

{
  "status": "ok",
  "host": "db1.example.com",
  "version": "1.0.0",
  "scripts": [...],
  "tags": { "env": "production", "role": "database", "location": "ams1" }
}

Tag keys and values are arbitrary strings. No reserved keys. An agent with no [tags] section returns "tags": {}.

Script schema sidecars

Each allowlisted script may have an optional schema sidecar describing its arguments, so tools that consume the API can type each script. The agent looks for a file beside the script with .schema.json appended:

/opt/ctrl-exec-scripts/check-disk.sh
/opt/ctrl-exec-scripts/check-disk.sh.schema.json   ← optional sidecar

The sidecar is a JSON object holding a JSON Schema for the script's arguments plus protocol-neutral metadata (description, and the behavioural flags read_only / destructive / idempotent):

{
  "version": "2",
  "description": "Check disk usage against a threshold",
  "read_only": true,
  "arguments": {
    "type": "object",
    "properties": { "threshold": { "type": "integer", "minimum": 1, "maximum": 100 } }
  },
  "argv": [ { "arg": "threshold" } ]
}

When present, the agent advertises it in the /capabilities response as schema plus a derived schema_version (the explicit version, or a content hash), and /discovery and /openapi-live.json carry it through. The MCP bridge consumes this to expose each script as a typed tool; it is also usable by a UI or for argument validation.

The sidecar is read only for allowlisted scripts, capped at 64 KiB, and reloaded on SIGHUP with the allowlist. A missing, oversize, or invalid sidecar is skipped (logged as ACTION=schema-skip) — the script stays callable, just untyped. The agent never executes or interprets the sidecar; it only advertises it. See docs/SCHEMA-SIDECAR.md for the full format, the version-identity rule, and the cross-host version-collision behaviour.

ctrl-exec-agent

The ctrl-exec-agent binary runs on each managed host. It serves the mTLS listener, handles pairing, and executes allowlisted scripts on request from the dispatcher.

Synopsis

ctrl-exec-agent <mode> [options]

Global options

--config <path>: Path to agent.conf. Default: /etc/ctrl-exec-agent/agent.conf.
--allowlist <path>: Path to scripts.conf. Default: /etc/ctrl-exec-agent/scripts.conf.
--port <n>: Override the pairing port (default 7444). Applies to request-pairing only; the serve port is set in agent.conf.

serve

Start the agent server. Listens on the port configured in agent.conf (default 7443) for incoming mTLS connections from the dispatcher.

ctrl-exec-agent serve

Under normal operation this is started and managed by the init system (systemd or procd). Run directly for debugging or on systems without a supported init system.

The agent picks up an edit to agent.conf or scripts.conf by itself: before it handles each connection it checks whether either file changed on disk (inode, size, modification or change time) since it last read them, and if so reloads both, logs ACTION=allowlist-reload REASON=changed, and handles the connection with the new files. The executor reads both files afresh for every run, so the agent and the executor always act on the same files and no reload command is needed after an edit. The check is two stat calls per connection; the reload happens in the serve loop's parent, so every later connection sees it. SIGHUP still reloads on demand (REASON=signal), together with the revocation list and the trusted-dispatcher map (which are reloaded with every reload, but a change to them alone is picked up only on SIGHUP):

# On systemd hosts (preferred)
systemctl reload ctrl-exec-agent

# On systems without systemd
kill -HUP $(pidof ctrl-exec-agent)

agent.conf is parsed into a new configuration first and swapped in only if it parses. If it does not, the previous agent.conf and allowlist stay in force and the reason is logged once (ACTION=reload-fail) - a reload never leaves the agent serving a half-applied or fail-open configuration. A scripts.conf entry naming a profile agent.conf does not define skips only that script (script not permitted: profile '<name>' is not defined in agent.conf); the rest loads. On a successful reload allowed_ips, the rate-limit settings, max_children, auth_hook, script_dirs, [tags], [profile ...] blocks, executor_socket and the other keys apply to subsequent connections. port, cert, key and ca cannot change without a restart (the listener is bound and holds that identity): the running values are kept and ACTION=reload-note logs that a restart is needed. The executor re-reads agent.conf by itself when its modification time changes.

request-pairing

Submit a pairing request to a dispatcher host. Generates a key and CSR for this agent, connects to the dispatcher's pairing port (7444), and waits for the operator to approve the request. The agent's configured operational port (port in agent.conf, default 7443) is reported in the request and stored in the registry so dispatch can reach it without a per-command --port; the operator can override it with approve --agent-port.

ctrl-exec-agent request-pairing --dispatcher <host> [--dispatcher-fingerprint SHA256:<hex>] [--rekey]
ctrl-exec-agent request-pairing --dispatcher <host> --background [--timeout <n>]
    [--dispatcher-fingerprint SHA256:<hex>] [--rekey]

An agent that is not yet paired generates a new key. An agent already paired keeps its key and certificate; see Pairing with another dispatcher below.

--dispatcher <host>: Hostname or IP of the dispatcher host. Required.
--dispatcher-fingerprint SHA256:<hex>: Verified pairing. The SHA-256 fingerprint of the dispatcher's CA certificate, from ced ca-fingerprint or the pairing-mode banner. Colons between the bytes and either case are accepted, and the SHA256: prefix may be left off; anything else is refused before connecting. After the TLS handshake, and before sending anything, the agent checks that the dispatcher's certificate chains to the CA with this fingerprint. If it does not, the agent stops with Pairing refused: and the expected and seen fingerprints, logs ACTION=pair-verify-failed, and exits non-zero without sending its CSR (with --background, before printing a request ID). It also stops, the same way, when the certificate was issued by that CA but is marked as an agent certificate (0.14): whatever answered holds an agent's key, not the dispatcher's. A dispatcher certificate issued before 0.14 (unmarked) is accepted. On success it prints Dispatcher verified: ... above the pairing code. Use it on any network you do not control; see SECURITY.md, Verified pairing. Needs a 0.14 or later dispatcher, which sends its CA certificate. The CA certificate the dispatcher delivers at approval must be that same CA, and only it; otherwise nothing is stored and the agent stops with Pairing refused: (ACTION=pair-verify-failed). Without the flag, pairing runs as before.
--rekey: Generate a new key and have this dispatcher sign it, even when the agent is already paired. The new certificate replaces the one the other dispatchers verify, so each of them refuses the agent until it is re-paired with it; the command names them. Use it when the certificate's issuing dispatcher is gone (below).
--port <n>: Override the pairing port on the dispatcher (default 7444).
--lookup-by <ip|hostname>: Suggest how the dispatcher should resolve this agent for dispatch (default: hostname). Sent with the pairing request; the operator's approve --lookup-by overrides it. See Dispatch host resolution under ced run.
--background: Non-interactive mode for orchestrated installations. Prints the request ID to stdout and exits immediately, leaving a background process to wait for approval and store the certificate on receipt. See Orchestrated pairing below.
--timeout <n>: How long (in seconds) the background process waits for approval before giving up. Default: 30. Maximum: 600. If the timeout expires the background process exits with code 2 and logs ACTION=pair-timeout. Only valid with --background.

The command blocks until the dispatcher approves or denies the request, or until the connection times out. On approval, the signed certificate and CA certificate are written to the config directory and the agent is ready to serve.

If the connection fails with a configuration error, check that the dispatcher host is reachable, that pairing-mode is active on the dispatcher, and that the correct address was specified.

Pairing with another dispatcher

An agent already paired with dispatcher A pairs with dispatcher B, which has its own CA, with the same two steps as any pairing: request-pairing --dispatcher B on the agent, approve on B. No key or CA file is copied between hosts.

The agent finds it is paired: agent.crt, agent.key and ca.crt are present, the certificate verifies against the CA in the bundle that issued it, and its key is agent.key. It prints Already paired: keeping this agent's key and certificate (issued by CA SHA256:...).
It sends a CSR made from its existing key, with its certificate and that CA certificate. The pairing code covers all three. B checks that the CSR and the certificate carry the same key, the CSR's self-signature, and that the certificate verifies against the CA sent; anything else is refused before it is queued. B's acknowledgement confirms it keeps the certificate.
The operator compares the code and the two fingerprints (list-requests) and approves on B. B signs nothing; it pins the agent's issuing CA in that agent's registry entry and returns its own CA certificate and identity.
The agent adds B's CA to its bundle and B to its trusted-dispatcher map. agent.crt and agent.key are unchanged, so A keeps verifying the agent. The running agent sees ca.crt and the map change and reloads them, the TLS listener's CAs included (ACTION=ca-bundle-reload), at its next connection: no restart.

Renewal stays with the dispatcher that issued the certificate. B pins the CA, not the certificate, so A's renewals do not affect B; B never renews it.

A dispatcher older than 0.14 does not confirm that it keeps the certificate. The agent then withdraws before anything is stored (Pairing not done: ..., ACTION=pair-keep-unsupported); deny the request on that dispatcher. Upgrade it, or use --rekey.

Re-pairing with the dispatcher that issued the certificate also keeps it: that dispatcher registers the agent as its own, pins nothing, and goes on renewing it.

Two CAs with one name: Before 0.14 every dispatcher CA was named ctrl-exec CA, and a dispatcher certificate issued before 0.14 on an OpenSSL 1.1 host has no Authority Key Identifier. An agent trusting two such CAs finds the issuer of that certificate by name alone, checks it against the first CA of the name in ca.crt, and refuses it at the TLS handshake when the other CA issued it. The refusal is correct (nothing is accepted that does not verify) and stays. The agent logs it as ACTION=tls-failure with a REASON ending in "the dispatcher certificate (serial ...) predates 0.14: ... so it is not compatible", and with the fix; the dispatcher's error for that agent says its certificate predates 0.14; doctor notes it on both sides (ca-same-name, dispatcher-cert-no-aki). Upgrading the dispatcher keeps its old certificate. The order that avoids the refusal: on that dispatcher, upgrade to 0.14 or later and run ctrl-exec-dispatcher rotate-cert first, then pair the agent with it; the new certificate carries the identifier and the agent accepts it. If the agent is already paired and refusing it, do the same and then pair the agent with that dispatcher again (request-pairing --dispatcher B, approve on B): the rotation broadcast cannot reach an agent that refuses the certificate it is sent with, so only pairing records the new serial. See TROUBLESHOOTING.md.
When the issuing dispatcher is gone: An agent whose certificate A issued, with A removed (remove-dispatcher) or decommissioned, can no longer have its certificate renewed: only A's CA may renew it (the agent refuses a renewal from another CA, so no dispatcher can move it onto its own CA and cut the others off). Before the certificate expires, re-key with one of the remaining dispatchers, B: request-pairing --dispatcher B --rekey, approve on B. B now issues and renews the certificate. Every other remaining dispatcher pinned A's CA for this agent and refuses the new certificate, so re-pair each of them (request-pairing --dispatcher C, no --rekey): the agent keeps the B-issued certificate and C pins B's CA. Until then C cannot reach the agent. With B the only remaining dispatcher there is nothing more to do.

Orchestrated pairing

For automated provisioning workflows where interactive approval is not possible, --background separates the pairing request submission from the approval wait. The foreground process exits as soon as the dispatcher acknowledges the request, printing the request ID to stdout. A background process holds the connection open and writes the certificate when approval arrives.

The orchestrator's responsibility is to capture the request ID and call ced approve on the dispatcher host before the timeout expires.

Flow

On the agent host (as part of a provisioning script):

# Start pairing-mode on the dispatcher first, then:
REQID=$(ctrl-exec-agent request-pairing --dispatcher ctrl-exec.example.com \
    --dispatcher-fingerprint "$DISPATCHER_FINGERPRINT" \
    --background --timeout 60)
echo "Request ID: $REQID"

DISPATCHER_FINGERPRINT is the output of ced ca-fingerprint, held in the orchestrator's configuration alongside the dispatcher's address. With it, a dispatcher that fails verification gets no CSR and the command exits non-zero with nothing on stdout.

The command exits 0 immediately, printing the request ID and nothing else. The background process is now waiting for approval.

On the dispatcher host (or via the orchestrator calling it remotely):

ced approve "$REQID"

The background process receives the certificate, writes it to /etc/ctrl-exec-agent/, logs ACTION=pair-complete, and exits 0.

Confirm pairing succeeded on the agent host:

ctrl-exec-agent pairing-status
# Exits 0 if paired, 1 if not yet paired

Exit codes (background process)

0: Approval received and certificates stored successfully.
2: Timeout expired before approval arrived. Key and CSR have been cleaned up. Re-run request-pairing to try again.
3: Request was explicitly denied by the operator.

Timing

The background process holds the connection open for up to --timeout seconds (default 30, maximum 600). The dispatcher's own polling window is 600 seconds — approval must arrive within whichever is shorter. For orchestrated workflows where the approval step may take time, increase --timeout accordingly:

REQID=$(ctrl-exec-agent request-pairing --dispatcher ctrl-exec.example.com \
    --background --timeout 120)

Notes

The background process inherits the connection socket from the foreground process — no reconnection occurs. The maximum --timeout of 600 seconds is enforced because the dispatcher closes the connection after its own 600-second polling window, making longer waits unreliable.

The request ID printed to stdout is the same ID shown by ced list-requests on the dispatcher host. Both the confirmation code and the request ID are available via list-requests for verification before approving.

pairing-status

Report whether the agent is paired and show the certificate expiry date.

ctrl-exec-agent pairing-status

Exits 0 if paired, 1 if not paired. Suitable for use in scripts and health checks.

The certificate directory is root-only. Run unprivileged, the command says so (pairing-status needs root - cannot read <file> (Permission denied), exit 1) rather than reporting the agent as not paired. The same applies to self-check and self-ping: a permission error on agent.conf, scripts.conf or a certificate file is reported as "needs root - run it with sudo", not as a configuration error. A real configuration error is reported as before.

remove-dispatcher

Stop trusting one dispatcher, without touching the others. Root only.

sudo ctrl-exec-agent remove-dispatcher <id>

<id> is the dispatcher identity shown by list-dispatchers. The command:

removes every trusted-map entry for that id (the map is written first, so a failure part-way never leaves trust extended);
removes each CA those entries were under from the CA bundle (ca.crt), unless an entry that stays is under it. A CA shared by two dispatchers stays until the last of them goes. Both files are written atomically;
sends SIGHUP to the running agent (found through --pidfile <path> if given, else the init script's /var/run/ctrl-exec-agent.pid, else systemd's MainPID; the process must be a ctrl-exec-agent), so the map change takes effect at once: the dispatcher's next connection is refused as not trusted. When no running agent is found it prints the reload command;
logs ACTION=remove-dispatcher ID=<id> ENTRIES=<n> CA=<fingerprints removed, or ->.

An unknown id changes nothing, exits 1 and names the known ids: unknown dispatcher 'x' (known: d1, d2). No id: usage, exit 2.

Reload. The agent follows ca.crt and the map without a restart (T-062): at the reload the map no longer trusts the dispatcher, and a CA removed from ca.crt leaves the TLS listener, so a certificate from it no longer completes the handshake. No restart is needed.

The dispatcher that issued the agent's own certificate. self-ping checks the agent's certificate against ca.crt, and renewal is pinned to its issuing CA, so that CA is never removed from the bundle. Removing the dispatcher still removes its map entries and the command prints and logs (REASON=agent-cert-issuer) WARNING: This agent's certificate was issued by '<id>', which can no longer renew it; it expires on <date>. Before that date, have one remaining dispatcher issue a new certificate (ctrl-exec-agent request-pairing --dispatcher <host> --rekey), then re-pair each other remaining dispatcher without --rekey so it pins the new issuing CA; see "When the issuing dispatcher is gone".

The last dispatcher. Removing the only remaining dispatcher is allowed. The map is then empty and the agent refuses every caller, but it still starts and answers: ca.crt is never left empty (an empty CA file would stop the TLS listener starting), so the last CA stays in the bundle and the command says so. self-check then prints Dispatchers: NONE trusted. Pair a dispatcher again with request-pairing to serve one.

list-dispatchers

List the trusted dispatchers. Root only.

sudo ctrl-exec-agent list-dispatchers

One line per dispatcher and CA, sorted by id then CA, fields separated by one space:

<id> <ca-fingerprint> <serial>[,<serial>...] [issuer-of-agent-cert]

The optional last field, issuer-of-agent-cert, is present only on a line whose CA issued this agent's own certificate (cert in agent.conf).

<ca-fingerprint> is the 64-hex SHA-256 of the issuing CA (the value ctrl-exec-dispatcher ca-fingerprint prints without the SHA256: prefix, in lowercase), or - for an old map line that could not be tied to a CA. <serial> is the certificate serial in lowercase hex; a dispatcher with a rotation overlap lists both. Nothing is printed when no dispatcher is trusted. self-check prints the ids on one Dispatchers: line, and an Agent certificate issued by: line naming the issuing dispatcher.

Confidential run input (scripts.conf `input=`)

A run request may carry input: an object of named string fields (names A-Za-z0-9_.-, 1-64 characters, at most 32 fields, at most max_input_bytes of values in total, default 1048576). Input is never in the script's argument list, never in a hook's environment or stdin (hooks get input_names and input_bytes only), never in a log line and never in the run store. The script's own scripts.conf entry decides how it receives the input, so a script changes form on the agent alone:

input=context (default): The context JSON on stdin gains an input object, only when the request had input. Existing scripts are unaffected.
input=raw or input=raw:<name>: Stdin carries the bytes of one field. raw needs exactly one field; raw:<name> needs the field <name> (other fields are ignored). A request that does not fit is refused before the script starts, naming the field (invalid input: script takes input=raw:key but input field 'key' is missing). The context JSON, without input and without the caller's token, is in CTRL_EXEC_CONTEXT. The environment is inherited by every program the script starts, so the token is kept out of it; a script that needs the token uses input=context.
input=fields: Stdin is empty. Each field is a file $CTRL_EXEC_INPUT/<name> (mode 0600) in a private per-run directory (mode 0700) which is removed when the script exits or is killed. CTRL_EXEC_CONTEXT is set as for raw. With the executor (executor_socket set), the executor creates the directory as root at /run/ctrl-exec-agent/input/<reqid>.<random> (the input directory beside the executor's socket) before it applies the script's profile, and hands the directory and its files to the profile's run_as user (the executor's own user, root, when the profile has no run_as). It is visible, read-only, inside a profile with writable. Without the executor the agent creates the directory as its own user under input_dir in agent.conf (default the system temporary directory, which under the shipped systemd unit is the service's private /tmp); the run user is therefore the agent user.

A scripts.conf entry with any other input= value (input=bogus, an empty value, a raw: name that is not a valid field name) is skipped, and a run of it reports script not permitted: its input mode '<value>' is not one of context, raw, raw:<name>, fields. /capabilities lists each script's input mode.

All three modes work with and without the executor. With the executor, the mode is taken from the executor's own reading of scripts.conf, as the path and profile are; the agent's reading only decides what it checks and sends first. The executor applies its own limits to what the agent hands it: at most 32 fields, names as above, and at most 4194304 bytes (4 MiB) of values in total, so a max_input_bytes above 4 MiB on the agent is refused at the executor (executor: input exceeds the executor's limit of 4194304 bytes). A script the executor reads as raw or fields is refused when the request carried no input: executor: scripts.conf declares '<name>' input=fields, but the request carried no input; if scripts.conf changed, run 'systemctl reload ctrl-exec-agent'. The agent sends a request without input only for a script its loaded scripts.conf has as input=context; since 0.13.2 it reloads an edited scripts.conf before the next request, so this is seen with an agent older than 0.13.2 that has not reloaded an edit, or one that predates input=. An agent newer than its executor refuses a run that needs input with ctrl-exec-executor does not support run input: it is older than this agent. Restart ctrl-exec-executor.service ...; runs without input, of input=context scripts, still go through it unchanged.

Confidential output (scripts.conf `output=`)

A script whose output is secret (an issuing script that prints a new private key, for example) is declared in the agent's scripts.conf with output=confidential, beside the path and any profile= or input=, in any order:

issue-cert = /opt/ctrl-exec-scripts/issue-cert.sh output=confidential

output=stored (default): The result is stored as before: the dispatcher's run store (/var/lib/ctrl-exec/runs/<reqid>.json, 24 hours) keeps stdout and stderr with the exit status.
output=confidential: The caller gets stdout and stderr in the synchronous response as usual, and the agent marks the result output_confidential: true. The dispatcher stores such a result without stdout and stderr (exit status, the marker and the other metadata only), so GET /status/<reqid> and ced status <reqid> show no output for it. No log line on the agent, the executor or the dispatcher carries a run's output, for any script. ced run prints the output to the operator and stores nothing for a synchronous run.

An async run of the script is refused before anything starts (script '<name>' is output=confidential: its output is never stored, so it cannot run async; run it synchronously): an async result has to be stored, on the agent until it is fetched and then in the dispatcher's run store, to be collected later.

A scripts.conf entry with any other output= value (output=secret, an empty value, output=Confidential) is skipped, and a run of it reports script not permitted: its output mode '<value>' is not one of stored, confidential. The executor reads the same value from its own scripts.conf and skips such an entry too. /capabilities lists each script's output mode.

The protection depends on the agent's declaration: the dispatcher cannot tell that output is secret unless the agent marks the result. A result without the marker is stored whole, which includes every result from an agent older than 0.13.0. Declare the script on an upgraded agent before relying on it. The caller (a ctrl-exec-cli script, a job's own log) is responsible for what it does with the output it receives.

Scripts limited per dispatcher (scripts.conf `dispatchers=`)

On an agent paired with several dispatchers, a script can be limited to some of them with dispatchers=, beside the path and any profile=, input= or output=, in any order:

rotate-keys = /opt/ctrl-exec-scripts/rotate-keys.sh dispatchers=d1
report      = /opt/ctrl-exec-scripts/report.sh      dispatchers=d1,d2

The value is one or more dispatcher ids from the agent's trusted-dispatcher map (/var/lib/ctrl-exec-agent/ctrl-exec-dispatchers, the id given at pairing), separated by commas with no spaces. Ids are compared exactly, case included. Certificate serials are not used, because they change on rotation. dispatcher= (singular) is read as dispatchers=, so that typo limits the script rather than leaving it open. If both spellings are on one line, the last one on the line wins, as with any repeated option. The agent and the executor apply the same rule.

Any other key=value option the agent does not know (timeout=5, schema=x, dispatch=d1) is ignored, as before. The agent logs allowlist-unknown-option for it once per load, and cea self-check shows [warning: unknown option '<key>'] next to the script. The entry is kept and runs as before.

No dispatchers= (default): Every trusted dispatcher can run and see the script, as before.
With dispatchers=: A listed dispatcher runs the script as usual. Any other trusted dispatcher does not see it in /capabilities (discovery), and a run of it is refused: script not permitted: dispatcher '<id>' is not in its dispatchers= list, logged as deny with the dispatcher. The dispatcher is the one the agent identified from the connection's certificate (issuing CA and serial, matched against its own trusted map), never anything the request says. With executor_socket set, the executor reads the same entry from its own scripts.conf and refuses a run for a dispatcher not listed (executor: script not permitted for dispatcher '<id>'), so an agent that has not yet picked up an edit cannot run the script for the wrong dispatcher.

An id that is not in the trusted map does not skip the entry: the agent logs allowlist-unknown-dispatcher once per load (start-up and each reload), and the script runs for that id once a dispatcher is paired under it. A value that is not a list of valid ids (empty, a trailing or doubled comma, a space after a comma, an id that does not start with a letter or digit or is longer than 64 characters) skips the entry: a run of it reports script not permitted: its dispatchers= value '<value>' is not a comma-separated list of dispatcher ids .... The executor skips the same entries.

cea self-check is local and lists every script, showing the limit ((profile: default, dispatchers: d1,d2)) and pointing out an id that no paired dispatcher has.

Auth hook (agent-side)

When auth_hook is configured in agent.conf, the hook is executed after every run request passes allowlist validation, before the script is spawned. It provides an independent authorisation layer separate from the dispatcher's own hook.

Context fields

The hook receives a JSON object on stdin:

{
  "action":     "run",
  "script":     "backup-db",
  "args":       ["--full"],
  "username":   "alice",
  "token":      "eyJ...",
  "source_ip":  "10.0.0.5",
  "input_names": ["key"],
  "input_bytes": 1704,
  "timestamp":  "2025-01-15T12:00:00Z"
}

input_names and input_bytes describe the run's confidential input: the field names and the total size of the values. The values themselves never reach a hook.

action is always "run" on the agent side. The agent does not call the hook for ping requests - those are handled before the hook path is reached.

Environment variables

The same context is available as environment variables:

ENVEXEC_ACTION: Always run on the agent.
ENVEXEC_SCRIPT: The script name from the allowlist.
ENVEXEC_ARGS: Space-joined argument string. Deprecated - lossy if arguments contain spaces. Prefer ENVEXEC_ARGS_JSON.
ENVEXEC_ARGS_JSON: JSON array of arguments. Reliable for all argument values.
ENVEXEC_INPUT_NAMES_JSON: JSON array of the run's input field names ([] when none).
ENVEXEC_INPUT_BYTES: Total size in bytes of the input values (0 when none).
ENVEXEC_USERNAME: Username passed by the dispatcher in the request body.
ENVEXEC_TOKEN: Token passed by the dispatcher in the request body.
ENVEXEC_SOURCE_IP: IP address of the dispatcher connection. Behind a reverse proxy this is the proxy's address, not the client's, so hooks must not use it for client identity there.
ENVEXEC_DISPATCHER: Stable identity of the calling dispatcher, resolved from its cert serial via the agent's trusted-dispatcher map. Author per-dispatcher policy against this value, never the serial - serials rotate, the identity does not. Empty when no dispatcher could be resolved.
ENVEXEC_DISPATCHER_SERIAL: The calling dispatcher's certificate serial (hex).
ENVEXEC_TIMESTAMP: ISO 8601 UTC timestamp when the hook was invoked.

Exit codes

0: Authorised. Script execution proceeds.
1: Denied - generic refusal. Logged as reason=denied.
2: Bad credentials - token not recognised or expired. Logged as reason=bad credentials.
3: Insufficient privilege - credentials valid but not permitted for this script or action. Logged as reason=insufficient privilege.

Any other non-zero exit code is treated as denied with reason hook exited N.

Differences from the dispatcher-side hook

The dispatcher-side hook (configured in ctrl-exec.conf) runs on the dispatcher host before dispatch and covers both run and ping actions across all target hosts. The agent-side hook runs on each managed host independently, covering only run requests for that agent.

A request passes both hooks before any script is executed. The hooks are independent - there is no shared state between them.

The agent hook does not receive a hosts field; the agent is unaware of which other hosts are targeted in the same dispatcher invocation.

If no hook is configured on the agent, the request is authorised unconditionally at the agent level. The agent relies on mTLS and the allowlist as its primary access controls; the hook is for supplementary policy enforcement such as token validation or time-of-day restrictions.

self-check

Validate the local configuration, allowlist, and certificates without making any network connections. Reports each check individually.

ctrl-exec-agent self-check

Checks performed:

agent.conf parses without error and required keys are present
scripts.conf allowlist loads and all listed scripts are executable; entries that were skipped (path outside script_dirs, relative, two paths, over-long, and with script_ownership = strict the ownership rule) are listed as SKIPPED with the reason
The script_ownership mode; with warn, a script that breaks the ownership rule is listed with [warning: <reason>]
Pairing certificates are present and not expired
The trusted dispatchers (Dispatchers: line); NONE trusted when the agent would refuse every caller
If script_dirs is configured, all scripts in scripts.conf fall under one of the permitted directories
If revoked_serials is configured, the file is readable and all entries are valid serial formats

Exit code is 0 if all checks pass. Use this after installation or configuration changes to confirm the agent will start cleanly.

self-ping

Perform a live mTLS loopback test against the running agent. Connects to 127.0.0.1:7443 using the agent's own certificate, verifies the port is listening, completes the mTLS handshake, and sends a /ping request.

ctrl-exec-agent self-ping

The agent will reject the connection with a 403 serial mismatch — the agent's cert is not a dispatcher cert and the connecting cert serial is not in the agent's trusted-dispatcher map. This is the expected and correct result. self-ping treats a 403 response as a pass: it confirms that the port is listening, the TLS stack is functioning, and the agent is correctly enforcing serial policy.

Typical output:

Checking config...
  Config OK (port 7443)
  Paired: yes (cert expires Jun 18 11:09:58 2028 GMT)
Connecting to 127.0.0.1:7443...
  Port 7443: listening
  mTLS handshake: OK
  Ping response: 403 serial mismatch (expected - agent is running and enforcing policy)
Self-ping complete.

Contrast with self-check: that mode validates configuration, allowlist, and cert files locally with no network connection. self-ping requires the agent to be running and paired and exercises the live network path.

If nothing is listening on the configured port, self-ping stops before any pairing or TLS check and says so plainly, exiting 1:

ctrl-exec-agent is not running (nothing is listening on port 7443).
Start it with:
    sudo systemctl start ctrl-exec-agent

The start command shown matches the host's init system (systemd, /etc/init.d, or ctrl-exec-agent serve). The check is a plain TCP connect to loopback, so it needs no root. self-check makes no network connection for its checks, but adds an informational Agent: running or Agent: ctrl-exec-agent is not running line.

Use self-ping after installation to confirm the agent is reachable and policy enforcement is active before the dispatcher attempts its first connection.

doctor

List every problem the agent's files show, each with the command that fixes it. self-check says what the agent loads; doctor says what is wrong with it and how to fix it, including what the agent only logs.

sudo ctrl-exec-agent doctor [--fix] [--json] [--brief]

One line per finding, problems first, then a summary line:

PROBLEM  Script 'backup': /srv/scripts/backup.sh is not executable, so a run of it fails. Fix: sudo chmod +x /srv/scripts/backup.sh
note     '/srv/scripts' is group-writable: 3 scripts run (cert-deploy, cert-check, svc-restart) with a warning (script_ownership = warn); script_ownership = strict would refuse them. Fix: sudo chmod g-w /srv/scripts, once nobody needs to write there (or leave it: the scripts run as they are)
ctrl-exec-agent doctor: 1 problem, 1 note.

Check	Finding
`agent.conf`	cannot be loaded, so the agent does not start (problem, `agent-conf`)
`scripts.conf`	cannot be read (problem, `scripts-conf`); an entry skipped, with the reason and its fix (problem, `script-skipped`); a script missing or not executable (problem, `script-missing`, `script-not-executable`); the ownership rule broken under `script_ownership = warn` - the script runs, `strict` would refuse it - one note per reason naming the scripts (note, `script-ownership`); an unknown option (note, `script-unknown-option`); a `dispatchers=` id no paired dispatcher has (note, `script-unknown-dispatcher`)
Profiles and executor	a script declares a profile while `executor_socket` is unset, so it is not applied (problem, `profiles-not-applied`); `executor_socket` set but the executor not answering, so every run fails (problem, `executor`)
Pairing	not paired (note, `not-paired`); the agent certificate expired (problem, `agent-cert-expired`); the agent not running (note, `agent-not-running`)
Trust	a trusted-map entry whose CA is not in `ca.crt` - the listener refuses that dispatcher (problem, `trust-ca-missing`); a CA in `ca.crt` no entry uses and that did not issue the agent's certificate (note, `trust-ca-unused`); pre-0.14 map lines not yet upgraded (note, `trust-map-old-format`); no dispatcher trusted (note, `trust-none`); the agent's certificate issued by a dispatcher no longer trusted (`remove-dispatcher`), which nobody can renew - a problem when it expires within 30 days, a note otherwise (`agent-cert-issuer-untrusted`); the certificate issued by no CA in `ca.crt` (problem, `agent-cert-issuer`); `ca.crt` with no readable CA (problem, `ca-bundle`); CAs in `ca.crt` with the same name, under which a dispatcher certificate issued before 0.14 is refused (note, `ca-same-name`; see Two CAs with one name under `request-pairing`)
Revocations	with several CAs in `ca.crt`, lines of `revoked-serials` that name a serial alone (the pre-0.14 form) and so revoke it under every CA (note, `revoked-any-ca`)

--fix also makes the changes that cannot lose data or widen access, and lists each as fixed:

pre-0.14 trusted-map lines are upgraded in place, exactly as the agent does when it starts: each keeps the CA(s) it was accepted under, so no dispatcher gains or loses access;
a CA in ca.crt that no trusted-map entry is under and that did not issue the agent's certificate is removed. No dispatcher can be served under it, so trust only narrows (a stranger's certificate from it no longer completes the handshake); pairing that dispatcher again brings it back. The last CA is never removed.

Nothing else is changed. In particular the ownership and permissions of script directories are reported, never changed: tightening /srv/scripts can stop the people who write there. A second --fix changes nothing.

--json prints one object on stdout and nothing else:

{"findings":[{"check":"script-not-executable","fix":"sudo chmod +x /srv/scripts/backup.sh","fixed":false,"severity":"problem","what":"..."}],"fixed":0,"notes":0,"ok":false,"problems":1,"program":"ctrl-exec-agent"}

ok is true when no problem remains; fixed counts the changes --fix made. --brief lists problems and fixes only, and counts the notes; with nothing to fix it prints one line. The package upgrade and install.sh run doctor --brief after an upgrade and print the result; they never use --fix, and its result never fails the install.

Each finding is also logged: ACTION=doctor-finding PROGRAM=ctrl-exec-agent SEVERITY=<problem|note> CHECK=<id> WHAT=<text> FIX=<command> (FIXED=1 when --fix made it), at WARNING for a problem, INFO otherwise.

Without root it reports what it could not read as a note (Could not read /etc/ctrl-exec-agent/agent.conf (permission denied), so nothing else was checked) and exits 0; members of the ctrl-exec-agent group can read everything it checks.

Exit code: 0 when no problem was found (notes may remain), 1 when one was, 2 on bad usage.

ctrl-exec-api

The ctrl-exec-api binary exposes the dispatcher's run, ping, and discovery operations as an HTTP REST API. It is installed as a systemd service (ctrl-exec-api.service) and listens on api_port (default 7445).

Start manually for testing:

ctrl-exec-api --config /etc/ctrl-exec-dispatcher/ctrl-exec.conf

It can also listen on a unix socket (api_socket) for same-host callers, alone (api_port = off) or beside the TCP port. TLS is enabled on the TCP port when api_tls_cert and api_tls_key are set in ctrl-exec.conf; plain HTTP is used when neither is. A listener configuration that is incomplete or contradictory (no listener, half a TLS pair, an unreadable TLS file, an unknown socket group) is refused at start-up with exit 78. See api_socket, api_port and api_tls_cert above.

Over the socket:

curl -s --unix-socket /run/ctrl-exec-api/api.sock http://localhost/health

Full endpoint documentation is in API.md. Summary:

GET /: Returns a JSON index of all endpoints and spec URLs. Auth applies as for all endpoints.
GET /health: Returns { ok: true, version: "..." }. Use for liveness checks.
POST /ping: Body: { hosts, username?, token? }. Pings all specified hosts in parallel.
POST /run: Body: { hosts, script, args?, username?, token? }. Runs an allowlisted script on all specified hosts. Returns results including a top-level reqid for status polling and syslog correlation.
GET /discovery or POST /discovery: Returns registered agents and their allowlisted scripts. Optional body: { hosts?, username?, token? }.
GET /status/{reqid}: Returns the stored result for a completed run. Results are retained for 24 hours. Returns 404 if the reqid is unknown or has expired.
GET /openapi.json: Serves the static OpenAPI 3.1 spec as installed.
GET /openapi-live.json: Generates and serves a dynamic spec augmented with live host and script enumerations from the registry and a capabilities scan.

HTTP status codes: 200 success, 400 bad request, 403 auth denied, 404 not found / unknown agent (a target host is not a registered agent), 409 lock conflict, 500 server error.

Syslog

Both binaries log structured key=value records to syslog under the daemon facility. The dispatcher logs under the tag ctrl-exec-dispatcher, the API server under ctrl-exec-api, and the agent under ctrl-exec-agent. Scripts themselves may log under any tag they choose.

All log entries use the ACTION=<name> field to identify the event type. The full action catalogue — fields, priorities, example lines, and alerting guidance — is in LOGGING.md.

Key quick-reference patterns for common operations:

ACTION=run EXIT=<n> SCRIPT=<name> TARGET=<host:port> RTT=<ms> REQID=<id>   (ctrl-exec-dispatcher)
ACTION=run EXIT=<n> SCRIPT=<name> PEER=<ip> REQID=<id>                      (agent)
ACTION=ping STATUS=ok|error PEER=<ip> REQID=<id> RTT=<ms>
ACTION=revoked-cert PEER=<ip> SERIAL=<hex>
ACTION=serial-reject PEER=<ip> REQID=<id>
ACTION=rate-block PEER=<ip> REASON=volume|probe
ACTION=ip-block PEER=<ip>

To correlate a dispatcher log entry with agent log entries, filter both sides by REQID:

grep 'REQID=a1b2c3d4' /var/log/syslog

See LOGGING.md for the complete action reference, field glossary, and alert pattern tables.

Managing long-running processes

When a script is expected to run longer than read_timeout, the dispatcher will report a timeout and return a non-zero exit, but the script continues running on the agent. There is no mechanism to cancel it remotely.

To run a long-lived process and retrieve its output later:

Have the script start the process in the background (e.g. with nohup or systemd-run) and return immediately with a job identifier or PID written to a known path.
Use a second allowlisted script to poll status or retrieve output by reading that path.
Raise read_timeout in ctrl-exec.conf if the script must complete within a single dispatcher invocation and the runtime is known and bounded.

The agent logs ACTION=run only when the script process exits. If the dispatcher times out before the script completes, no ACTION=run entry appears in the agent's syslog until the script eventually exits. An operator cannot determine from syslog alone that a script is currently running — only that it was started (from the dispatcher-side ACTION=run entry) and has not yet completed.

Getting started with examples

The ctrl-exec-demonstrator script exercises all core ctrl-exec capabilities from a single agent-side script: stdout and stderr capture, exit code propagation, argument passing, JSON context logging, and agent-side information. It is installed on every agent host by the installer and is disabled in scripts.conf by default.

To enable it, uncomment the entry in /etc/ctrl-exec-agent/scripts.conf on the agent host. The agent picks the edit up at the next request; no reload is needed.

Then run the script directly on the agent host to see all available subcommands and the exact ced invocations that exercise them:

/opt/ctrl-exec-scripts/ctrl-exec-demonstrator.sh

ctrl-exec - Installation and Operations

ctrl-exec-dispatcher

Synopsis

Global options

Environment variables

run

Asynchronous runs (--async)

Dispatch host resolution

status

wait

ping

pairing-mode

list-requests

approve

deny

ca-fingerprint

list-agents

edit-agent

setup-ca

setup-ctrl-exec

rotate-cert

maintain

serial-status

doctor

ctrl-exec.conf

Cert rotation settings

agent.conf

Rate limiting settings

Agent tags

Script schema sidecars

ctrl-exec-agent

Synopsis

Global options

serve

request-pairing

Pairing with another dispatcher

Orchestrated pairing

Flow

Exit codes (background process)

Timing

Notes

pairing-status

remove-dispatcher

list-dispatchers

Confidential run input (scripts.conf input=)

Confidential output (scripts.conf output=)

Scripts limited per dispatcher (scripts.conf dispatchers=)

Auth hook (agent-side)

Context fields

Environment variables

Exit codes

Differences from the dispatcher-side hook

self-check

self-ping

doctor

ctrl-exec-api

Syslog

Managing long-running processes

Getting started with examples

Asynchronous runs (`--async`)

Confidential run input (scripts.conf `input=`)

Confidential output (scripts.conf `output=`)

Scripts limited per dispatcher (scripts.conf `dispatchers=`)