Design paper

A protocol and systems design paper for pirate-shell

Protocol: psh/3 · Implementation: 0.1.0-dev · Date: 2026-11-11

Source baseline: 2bbce66fa54f10f29f4739be38b30e825501335c

Abstract

psh, short for pirate-shell, is an experimental Linux remote-shell system comprising the psh client and pshd daemon. It combines a mandatory hybrid X25519 + ML-KEM-768 ephemeral key exchange with paired Ed25519 + ML-DSA-87 host and user identities. DTLS 1.3 supplies record protection and its key schedule; a bounded reliable transport supplies ordered application delivery over UDP. Client address migration requires encrypted return-path validation. A privileged broker independently verifies user authorization and both client signatures before launching a process, while network parsing runs in an unprivileged, restricted worker. Privilege-drop and exec boundaries remove inherited host private-key memory from workers and session relays.

The central design choice is to make a narrow security policy mandatory: public-key-only remote authentication, no root login, complete-pair host trust, one session per connection, explicit resource limits, and no legacy crypto fallback. This paper describes the implemented wire protocol, authentication construction, process architecture, operational defaults, and recorded validation evidence. It compares these choices with contemporary OpenSSH without treating feature reduction or stronger individual parameters as proof of overall security. OpenSSH already provides post-quantum key exchange and, in current releases, hybrid post-quantum signatures. psh’s distinguishing policy is requiring its specific hybrid profile throughout every accepted connection. The implementation remains a prototype without a formal protocol proof or an external security audit.

Keywords: remote shell; post-quantum cryptography; hybrid authentication; DTLS 1.3; privilege separation; reliable UDP; client migration; bounded resources.

Contents

1. Introduction and design thesis
2. Scope, method, and reading conventions
3. Threat model and security objectives
4. Protocol stack and connection lifecycle
5. Cryptographic profile and post-quantum rationale
6. Identities, fingerprints, and trust establishment
7. Stateless admission and datagram routing
8. DTLS records, fragmentation, and resource policy
9. Reliable delivery above DTLS
10. Authenticated client migration
11. Application framing and mutual authentication
12. Process architecture and privilege boundaries
13. Session semantics and application flow control
14. Liveness, revocation, and complete process cleanup
15. Credential storage, configuration, and build policy
16. Comparing policy with contemporary OpenSSH
17. Resource limits as part of the protocol
18. Validation evidence and experimental results
19. Limitations and research directions
20. Design assessment

1. Introduction and design thesis

A remote-shell service crosses several trust boundaries at once. It accepts untrusted network input, handles valuable long-lived credentials, authorizes a local account, creates a terminal or pipes, and launches a program with that account’s privileges. Encryption alone addresses only part of this problem. The relationship between the network parser and process launcher is equally important, as are credential provisioning, host trust, flow control, and cleanup.

psh explores a deliberately narrow answer: a remote shell whose accepted connections all use one cryptographic and authentication profile, whose network workers cannot assert successful user authentication, and whose transport and session queues have explicit limits. The implementation supports interactive PTYs and remote commands through the account’s configured shell. It also supports client IP and port changes while preserving the same live connection and process.

Here, “more sane defaults” means fewer decisions that an administrator must get right to obtain this particular policy. Password authentication cannot be enabled. Root login cannot be enabled. A classical-only peer cannot negotiate a connection. A session cannot request agent forwarding or a second channel. Those choices reduce configuration states and protocol features; they also remove capabilities that other deployments need. Their suitability is a policy judgment, separate from whether the implementation correctly enforces them.

The project makes four concrete engineering contributions within that scope:

1. A fixed DTLS profile plus application authentication that binds paired classical and post-quantum identities to a connection and exact session request.
2. Independent privileged user authorization, restricted host signing, and exec boundaries that keep long-lived private keys out of network workers and relays.
3. A bounded reliable byte stream with authenticated delivery metadata and explicit validation before promoting a new client address.
4. Application input credit and supervisor-owned process lifetimes, so a blocked command or uncooperative relay does not define the cleanup boundary.

These are implemented design properties, not claims of novelty relative to all prior secure transports or remote-login systems.

2. Scope, method, and reading conventions

This paper describes the source baseline above. The implementation and constants in src/ and include/psh/ determine current behavior. The shorter protocol reference , authentication contract , and crypto policy provide companion specifications. Historical test results come from the validation record ; this paper reports them as existing evidence, not as new experiments performed while writing it.

All multi-byte Internet protocol integers are unsigned and big-endian unless otherwise stated. Sizes are bytes; KiB means 1,024 bytes. Concatenation is written ||. A quoted domain ending in \0 includes exactly one NUL byte. Network formats are encoded explicitly and contain no native C structure padding. Local IPC uses native structures on Unix sockets between matching executables and is not an interoperable network protocol.

Version numbers refer to different layers:

The retained PSH1 magic and authentication domain suffixes do not enable old application profiles. Neither psh/1 nor psh/2 is accepted. There is no SSH wire compatibility and no automatic conversion of older identities or host pins.

The current implementation intentionally excludes file transfer, SSH agents, forwarding, PAM integration, multiple sessions per connection, server migration, and detached reattachment. It targets Linux x86-64 and requires Linux 5.14 or newer with usable cgroup v2 lifecycle facilities.

3. Threat model and security objectives

3.1 Adversaries and trusted components

The network adversary may observe, drop, reorder, duplicate, inject, and redirect datagrams; spoof source addresses; and record encrypted traffic for later cryptanalysis. A future adversary may obtain a cryptographically relevant quantum computer. A malicious client may complete admission and transport establishment, then send malformed or excessive application requests. An authorized account may launch commands that block, fork, change sessions, or leave descendants behind.

The design also considers a compromised network worker. That worker can access its own transport secrets and plaintext, interfere with its connection, and exercise the IPC capabilities made available to it. It must still present valid user proofs to the broker before a session can be launched. A worker compromise therefore remains serious: privilege separation constrains its authority but does not preserve the confidentiality or integrity of that worker’s session.

Trusted components include the kernel, privileged supervisor and authentication broker, administrator-controlled policy and credentials, the client endpoint, and the vendored cryptographic implementation. Root compromise and a malicious kernel are outside the protection provided by this architecture. An attacker who steals both private identity components can authenticate as that identity.

3.2 Intended properties

Traffic analysis is not prevented. IP addresses, packet timing, packet lengths, the outer packet type, and a stable connection identifier remain observable. Host provisioning is an external trust requirement. A first-use confirmation without independently checking the fingerprint remains vulnerable to an initial man-in-the-middle attack.

4. Protocol stack and connection lifecycle

The transport stack separates cryptographic record protection from ordered application delivery:

PTY / command input, output, signals, exit status
                    |
         psh session frames (type + length)
                    |
      bounded reliable stream (DATA / ACK / PROBE)
                    |
 encrypted record discriminator and migration controls
                    |
   DTLS 1.3: hybrid exchange, record AEAD, KeyUpdate
                    |
       PSH1 routing envelope and connection ID
                    |
                   UDP

DTLS preserves datagram semantics rather than providing a reliable application stream; psh implements that stream above it. The underlying protocol is specified in RFC 9147 . Raw-public-key TLS authentication is described in RFC 7250 .

sequenceDiagram
    participant C as psh client
    participant S as Root supervisor
    participant W as Network worker
    participant B as Privileged broker
    C->>S: INIT(CID, padding)
    S-->>C: COOKIE(expiry, address-bound MAC)
    C->>S: CONNECT(cookie)
    S->>B: Admit in private cgroup
    B->>W: Drop worker privileges and exec
    S-->>C: ACCEPT
    C->>W: DTLS 1.3 via supervisor
    W->>B: Restricted Ed25519 TLS signing
    B-->>W: CertificateVerify signature
    W->>B: HOST_AUTH_BEGIN(exporter)
    B-->>W: Constructed ML-DSA-87 HOST_PROOF
    W-->>C: HOST_PROOF
    Note over C: Verify proof and trust complete host pair
    C->>W: OPEN(exact session request and client pair)
    W->>B: AUTH_BEGIN(exact OPEN)
    Note over B: Authorize complete client pair
    B-->>C: CHALLENGE via worker
    C->>B: Dual client PROOF via worker
    Note over B: Verify both signatures, then launch
    B->>B: Drop account privileges and exec relay
    B-->>C: READY(8192 input credit) via worker
    C->>W: Session frames

The diagram omits retransmissions and error paths. Admission, DTLS completion, host trust, client authorization, and process readiness are distinct events. An ACCEPT does not authenticate the host. A completed DTLS handshake does not authorize the user. READY follows successful process launch and relay setup.

5. Cryptographic profile and post-quantum rationale

5.1 Mandatory algorithms

Tickets, resumption, early data, compression, and legacy protocol fallback are disabled. Both endpoints check the actually negotiated group before treating DTLS as complete or exporting authentication material. Failure to configure or negotiate the required group terminates setup; the implementation does not retry with a classical-only group.

All runtime primitives are supplied by pinned OpenSSL 4.1.0. ML-DSA uses Pure ML-DSA-87, an empty context, and normal message encoding. External-mu and test-entropy modes are not used. The standards for the lattice primitives are NIST FIPS 203 and NIST FIPS 204 .

5.2 Key establishment protects traffic confidentiality

X25519 provides the classical ephemeral Diffie–Hellman component. ML-KEM-768 provides the post-quantum encapsulation component. In the pinned OpenSSL implementation, the hybrid shared secret is the 32-byte ML-KEM secret followed by the 32-byte X25519 secret. OpenSSL feeds the hybrid result into the DTLS key schedule; psh does not implement a second combiner or its own traffic-key KDF. See the vendored hybrid algorithm documentation .

The intended benefit is resilience if one agreement component becomes weak while the other remains secure, subject to the hybrid combiner and protocol assumptions. A future quantum attack on X25519 alone should therefore not reveal the recorded session secret while ML-KEM remains secure. Conversely, the classical component hedges against a failure confined to the new KEM.

The ML-KEM-768 encapsulation key is 1,184 bytes and its ciphertext is 1,088 bytes. Adding the 32-byte X25519 contribution produces a 1,216-byte client key share and a 1,120-byte server contribution before handshake framing. These sizes explain why DTLS fragmentation is necessary under psh’s 1,200-byte UDP ceiling. The primitive sizes are specified in FIPS 203, Table 3 .

Fresh ephemeral agreement is distinct from signing with a long-lived identity. The intended forward-secrecy property is that later disclosure of identity keys alone does not reconstruct old traffic secrets after ephemeral secrets are erased. Endpoint compromise that captures live traffic secrets falls outside that claim.

5.3 Signatures protect authentication

Protecting a recorded conversation does not by itself prevent a future attacker from impersonating a server or user. psh addresses that separate objective with mandatory paired identities. An identity consists of one Ed25519 key and one ML-DSA-87 key, provisioned and compared as a complete pair.

For users, both signatures cover exactly the same transcript and both must verify. For hosts, Ed25519 possession is proved in the DTLS handshake and the ML-DSA signature authenticates a connection-bound application transcript that includes the complete host-pair digest. The client uses its actual DTLS peer key and locally derived exporter in verification.

The intended authentication hedge is conditional: if one signature scheme fails, the other remains mandatory, assuming connection binding, request hashes, trust provisioning, and the rest of the construction continue to hold. This is an engineering argument, not a formal proof of the application composition. The host construction is also deliberately asymmetric; it is not a native composite TLS signature algorithm.

ML-DSA-87 is the largest ML-DSA parameter set, with 2,592-byte public keys and 4,627-byte signatures. Its selection incurs larger messages and more work. It does not elevate every component to the same security level: ML-KEM-768, X25519, record authentication, hashes, and protocol composition have their own limits. This paper makes no whole-protocol category 5, FIPS-validation, or CNSA compliance claim.

Requiring two signatures is not multi-factor authentication. The two private components are stored together and generally unlocked by the same passphrase. The construction diversifies algorithm assumptions rather than establishing independent possession, knowledge, or hardware factors.

5.4 KeyUpdate and its limits

The client requests DTLS KeyUpdate for both traffic directions after one hour or 1,048,576 authenticated application records sent and received combined, as observed by the client. Server records lost before reception do not advance this trigger, so it is not a strict per-server encryption-operation ceiling under loss. Retransmitted reliable packets are new application records and contribute to this counter. The server responds without independently scheduling requests, avoiding overlapping requested updates that the pinned OpenSSL DTLS state machine rejects. Only the client side accepts a requested response before its preceding ACK; that response proves the server received the request even if its ACK was lost. Pending TLS exchanges and write retries complete before another request. Traffic-key updates do not need the host signing capability after authentication.

This coordination policy requires updating both endpoints: an older server still schedules potentially overlapping updates, while an older client lacks the combined record trigger.

KeyUpdate derives new traffic secrets from existing secrets. It performs no fresh X25519/ML-KEM exchange and does not recover confidentiality after disclosure of the current traffic secret. It should not be described as periodic new post-quantum agreement or post-compromise recovery.

DTLS can temporarily block application writes while awaiting a flight ACK. psh preserves one exact pending plaintext buffer, its length, and its destination across SSL_write_ex retries. Later emissions may be dropped as bounded local loss; they never overwrite the pending retry. Reliable transport and control timers provide subsequent recovery.

6. Identities, fingerprints, and trust establishment

6.1 Complete-pair representation

The public identity bytes are:

I = Ed25519_public[32] || ML-DSA-87_public[2592]       # 2624 bytes
H = SHA512("psh-identity-ed25519-mldsa87-v1\0" || I)  # 64 bytes
fingerprint = "SHA512:" || Base64_without_padding(H)

The displayed fingerprint has 93 characters. The algorithm order, fixed widths, and domain are part of the profile. Changing either component changes the identity. A broker must match all 2,624 public bytes from one authorized file; it cannot combine a matching Ed25519 component from one file with an ML component from another.

A private identity file contains exactly two PKCS#8 PEM objects, Ed25519 first and ML-DSA-87 second. Its .pub companion contains exactly two SPKI PEM objects in the same order. Only whitespace may precede, separate, or follow the objects. Missing, additional, reordered, duplicate, unsupported, partially decoded, or mixed encrypted/plaintext components are rejected.

The parser requires full ASN.1 consumption at the envelope, decrypted PKCS#8, and relevant embedded key layers. Accepting a valid prefix with ignored suffix bytes would create ambiguous credential encodings; current validation explicitly tests these cases. These files and fingerprints are distinct from OpenSSH’s formats.

6.2 Host trust policy

After receiving HOST_PROOF, the client first verifies the connection-bound cryptographic proof. It then requires a matching stored fingerprint, a matching provisioned complete public identity, or explicit first-use confirmation on a controlling terminal. Unknown hosts fail in batch mode unless provisioned through the supported trust path. Changed keys fail even if a new public key has been supplied; replacing an existing pin requires deliberate removal first.

Cryptographic possession and naming trust answer different questions. A peer that supplies two fresh keys and valid signatures has proved possession of those keys, but has not established that it is the intended machine. The user must verify the complete fingerprint through a trusted channel before first approval. No client identity, username, command, or session input is sent before host-pair verification and trust acceptance.

The trust store contains newline-terminated records:

lowercase-host<TAB>decimal-port<TAB>SHA512:unpadded-base64<LF>

Its parser rejects malformed or duplicate records anywhere in the file, invalid fingerprints including noncanonical Base64 padding bits, unsafe files, and limits above 64 KiB or 1,024 entries. Legacy SHA256: records can be diagnosed and removed but never authenticate a psh/3 identity. Trust is indexed by normalized host text and port, rather than treating DNS resolution as authentication.

6.3 Concurrent trust updates

A protected .lock sidecar supplies a stable locking inode even when the store is atomically replaced during deletion. Holding a lock only on the old store inode would allow a waiting writer to modify a file no longer published at the store path.

The client releases locks while waiting for human confirmation. After approval, it reacquires the sidecar lock, reopens the current store, and revalidates contents, ownership, permissions, and limits. A concurrent identical pin can be reused; a conflicting pin causes rejection. Unrelated additions and deletions are preserved. This keeps a slow trust prompt from blocking unrelated pinned logins.

7. Stateless admission and datagram routing

7.1 Outer envelope

Every UDP payload is at most 1,200 bytes and begins with this 22-byte envelope:

The CID is client-generated random routing material, not a credential or bearer capability. The envelope is outside DTLS encryption. Knowing or changing a CID does not create valid DTLS records. The CID is additionally bound into exporters and authentication transcripts; it is not treated as authenticated merely because it appears in an envelope.

7.2 Cookie admission

INIT elicits a same-size COOKIE response without allocating a per-connection worker. The expiry uses the server’s monotonic seconds and a 30-second window. Its MAC binds the CID, expiry, peer address family, source address and port, IPv6 scope, and contacted local destination address/zone. The client treats the cookie as opaque. A CONNECT must echo a valid unexpired cookie before allocation.

The HMAC uses a random 32-byte daemon secret. The comparison is constant-time. The fixed equal-sized initial exchange limits pre-admission reflection, while the cookie demonstrates source return reachability. It does not identify or authorize the connecting user. A reachable attacker can still consume admission capacity; the cookie is not a complete denial-of-service defense.

The supervisor admits at most 16 pending authentications and 64 total connection slots. Retiring connections retain slots until their lifecycle cleanup finishes. Repeated valid CONNECT packets for an existing routing tuple do not launch a second session. The initial client admission loop retries at 250 ms intervals and has a 15-second deadline.

7.3 Local endpoint integrity

The root supervisor owns all UDP listeners. It obtains each datagram’s contacted local address from checked IPv4/IPv6 packet-info ancillary metadata. Missing, truncated, or invalid metadata is rejected. Replies use that same local source address, including on wildcard listeners.

Routing checks the listener, local endpoint, and CID. Only link-local IPv6 addresses retain an interface zone; global destinations leave egress routing free to follow client movement. Workers can nominate remote candidate addresses but cannot change their connection’s local server endpoint. Server-address migration is not supported.

The supervisor validates fixed framing and relays ciphertext through bounded Unix datagram IPC. It does not parse TLS handshake messages. The worker uses datagram memory BIOs and explicit source metadata rather than allowing a socket library to update the peer address as a side effect of packet receipt.

8. DTLS records, fragmentation, and resource policy

OpenSSL performs handshake fragmentation and reassembly under a 1,178-byte DTLS MTU, leaving 22 bytes for the envelope. Large key shares therefore do not increase the UDP ceiling. The DTLS handshake has a 15-second deadline, and input/output datagram BIO rings are each limited to 65,536 bytes including their internal queue metadata.

The configured certificate-list limit is 16,384 bytes. In the pinned no-compression OpenSSL build, the resulting generic handshake-message ceiling is 16,716 bytes, not exactly 16 KiB. Future-message buffering is limited to ten messages ahead, and DTLS ACK retention to 1,023 record numbers. These are dependency-derived properties that must be checked again when updating OpenSSL. They do not represent an aggregate process-memory cap.

Encrypted application records begin with one discriminator byte:

Reliable headers, acknowledgment numbers, advertised receive credit, and migration tokens are all inside DTLS record protection. Network attackers do not get an unauthenticated ACK or flow-control side channel through the outer envelope.

An active peer can always stop delivering useful traffic. Bounds limit the state and time committed to that peer; they do not promise availability under bandwidth exhaustion, sustained CPU pressure, or all valid-credential attacks.

9. Reliable delivery above DTLS

9.1 Header and acknowledgment semantics

The reliable transport turns encrypted datagrams into one ordered byte stream per direction. Each packet has a 36-byte header:

Sequence numbers count segments, starting at zero, rather than byte offsets. All segments below the cumulative ACK are acknowledged. SACK bit i describes sequence ack + i; bit zero must be clear because the cumulative ACK identifies the first missing segment. DATA must be nonempty. Sequence-space exhaustion fails rather than wrapping into reused sequence numbers.

There are 64 transmit and 64 receive slots per direction. Each slot holds at most 1,024 bytes, yielding 65,536 bytes of payload capacity per side of the reliable queue. Receive credit is encoded as available slots multiplied by 1,024; it is not a raw slot count. Partially read receive segments retain their slots until consumed.

Parsing validates exact length, version, type, reserved fields, credit alignment and range, ACK/SACK references to actually sent data, and sequence constraints before mutating state. Valid out-of-window DATA can be discarded and acknowledged. Duplicates do not repeat byte delivery. Pure ACKs do not generate ACK loops. When a receive-window update is lost, bounded PROBE traffic requests a fresh ACK.

9.2 Replay protection and retransmission

Reliable-delivery sequence numbers and DTLS record numbers serve different purposes. A lost DATA segment is retransmitted with the same delivery sequence inside a fresh DTLS application record. Cached ciphertext is never replayed as the reliability mechanism. DTLS record replay checks and reliable duplicate suppression can therefore coexist.

The stream delivers bytes once and in order within a live connection, with partial reads permitted. It does not provide an exactly-once execution guarantee across reconnects or process crashes. If a command executes and its EXIT is lost before disconnection, a new invocation cannot infer whether retrying the command is safe. psh has no cross-connection operation identifier or durable execution log.

9.3 Congestion control and flow control

The initial congestion window is four segments. The implementation uses Reno-style slow start and additive growth, loss reduction, paced sending, smoothed RTT and variance, Karn-style avoidance of ambiguous retransmission samples, and fast loss detection after three newly acknowledged higher segments. Retransmission timers begin at one second and back off to at most 60 seconds.

Receiver credit and congestion window constrain transmission separately. The fixed 64-slot window also limits throughput on paths with large bandwidth-delay products. This is a compact congestion controller, not evidence of TCP-equivalent fairness or competitive bulk-transfer performance under arbitrary Internet load.

All session frame types share the ordered stream in their direction. Packet loss can therefore delay later control frames behind an earlier missing segment. The design preserves control progress under blocked process input; it does not eliminate ordered-stream head-of-line blocking.

10. Authenticated client migration

Client movement changes a UDP source address or port without creating a new connection or launching a new shell. The CID locates the existing connection, but migration is allowed only after the DTLS handshake and requires more than successful CID lookup.

1. A successfully authenticated DTLS record from a different source establishes a candidate path, with a fresh unpredictable 32-byte token.
2. The worker sends an encrypted PATH_CHALLENGE to that exact candidate.
3. Reliable data and ACK metadata from the candidate are discarded while its return reachability remains unproved.
4. A matching encrypted PATH_RESPONSE from the candidate promotes it.
5. Congestion and RTT estimates reset for the new path. The migration congestion window starts at one segment rather than inheriting the old path’s capacity.

Candidate state expires after three seconds when evaluated and may be replaced. Before promotion, normal application writes still target the validated peer. Candidate replies are limited to three times the authenticated received bytes credited to that candidate, including the outer envelope in sent-byte accounting. A datagram containing several records earns its byte credit only once.

Successful AEAD verification proves that a record belongs to the connection; it does not prove that its apparent source can receive replies. The token round trip supplies that additional check. An attacker who redirects a captured record must not thereby cause permanent peer-address promotion or acceptance of its reliable metadata.

There is a subtle interaction with DTLS KeyUpdate: authenticated handshake or ACK records may arrive from the new address while application writes are blocked. The worker detects authentication through a pinned OpenSSL callback that occurs after AEAD and replay checks, and can route the necessary DTLS control replies to the candidate under the same amplification budget. Buffered application plaintext whose source becomes ambiguous is dropped for reliable retransmission rather than attributed to the most recently received datagram.

SIGUSR1 requests a new UDP socket in the client and prompts path validation. That provides an explicit source-port rebind. A new psh invocation starts a new session; there is no detached reattachment token or server migration protocol.

11. Application framing and mutual authentication

11.1 Frame envelope and registry

Every application frame is encoded as:

type[1] || payload_length[4, big-endian] || payload[payload_length]

The maximum payload is 11,015 bytes. Large proofs and OPEN requests span reliable segments; they are never oversized UDP datagrams. The receiver checks type, length, available storage, expected phase, and type-specific contents.

One mutual authentication and one OPEN are permitted per connection. Repeated requests and out-of-phase messages terminate authentication rather than restart it. Failure diagnostics are not credentials; the client does not reinterpret a failure as authorization to fall back to another authentication method.

11.2 Exporter and host proof

DTLS exports 32 bytes using label EXPORTER-psh-auth-v2 without NUL and the 16-byte CID as exporter context. Denote those bytes by E and the complete host-pair digest by H.

T_host = "psh-host-auth-v1\0" || H[64] || CID[16] || E[32] || N_host[32]
         # exactly 161 bytes

HOST_PROOF = host_ML_public[2592] || N_host[32]
             || ML-DSA-87.Sign(host_ML_private, T_host)[4627]
             # exactly 7251 bytes

The privileged broker generates N_host and constructs every transcript field from its loaded host identity, supervisor-supplied CID, and the immutable exporter assertion. It does not accept an arbitrary worker-supplied message to sign with ML-DSA.

The client reconstructs the paired identity from the Ed25519 public key actually used by this DTLS peer and the ML public bytes in HOST_PROOF. It computes H and verifies the ML proof against its own CID and exporter. A valid proof then proceeds to complete-pair trust verification. The client receives no TLS client certificate request as its application user-authentication mechanism.

11.3 Exact OPEN encoding

The maximum OPEN is 10,887 bytes. Exact length is required, with no trailing bytes. Username and command cannot contain NUL. Username characters are limited to ASCII letters, digits, underscore, hyphen, and dot. PTY dimensions are validated before allocation; an empty command requests the account’s login shell.

The client sends these bytes only after completing host verification and trust. The worker forwards the exact encoding to the broker rather than replacing it with a native decoded structure for signature binding.

11.4 Broker challenge and dual client proof

The broker decodes OPEN, independently loads the account authorization policy, and verifies that its complete client pair matches one protected authorized file. It then generates a fresh 32-byte nonce N_client. The client and broker independently construct:

T_client = "psh-client-auth-v2\0"
           || H[64] || CID[16] || E[32] || N_client[32]
           || SHA512(exact_OPEN_payload)[64]
           # exactly 227 bytes

PROOF = Ed25519.Sign(client_Ed_private, T_client)[64]
        || ML-DSA-87.Sign(client_ML_private, T_client)[4627]
        # exactly 4691 bytes

The OPEN hash covers its payload, including the fixed prefix and all public identity, account, terminal, and command bytes. It does not hash the five-byte application frame envelope. Fixed-size fields and explicit domain separation avoid ambiguous concatenations and cross-purpose reuse.

The broker independently verifies both signatures. A successful transport handshake, worker assertion, matching public key alone, or one valid signature cannot authorize process launch. The nonce is single-use, with one proof attempt and a 15-second challenge deadline. The broker marks the attempt consumed before verification. Mutation of the account, command, PTY request, client pair, host pair, CID, exporter, or nonce invalidates the intended binding.

11.5 Irreversible broker phases and exporter boundary

The broker’s authentication state progresses through:

TLS_SIGNING -> HOST_ISSUED -> CLIENT_CHALLENGED -> PROOF_USED

During TLS_SIGNING it permits at most two Ed25519 server CertificateVerify operations. Each request must be exactly the TLS 1.3 server-signing structure: 64 spaces, TLS 1.3, server CertificateVerify, one NUL, and a 32-byte SHA-256 transcript digest. Both the worker signing provider and the broker validate that structure.

HOST_AUTH_BEGIN is accepted once, requires a preceding TLS signature, freezes one exporter, and closes the TLS-signing phase permanently. It permits at most one ML host proof. Later AUTH_BEGIN contains only exact OPEN bytes and cannot replace the exporter. No phase can be reset by sending another OPEN or asking for another challenge.

The qualification is essential: the broker does not independently verify TLS completion or derive the exporter. The exporter remains a worker assertion; a signing count is not proof of a completed handshake. This boundary preserves independent user authorization and constrains signing capability, but it does not make a compromised TLS endpoint trustworthy. Such an endpoint can observe or alter the traffic it terminates even when it cannot forge an unapproved client’s required signatures.

The broker’s initial 60-second budget includes the host proof and waiting for OPEN. A challenge gets a new local 15-second deadline, subject to the supervisor’s fixed 65-second admission-to-authenticated limit. A late challenge can therefore have less than 15 seconds left. Incoming traffic does not extend that outer deadline. The client’s host-proof receive deadline is 15 seconds; its later CHALLENGE/READY waits use 20-second local receive budgets.

12. Process architecture and privilege boundaries

flowchart TD
    S["Root supervisor: listeners, configuration, host pair, lifecycle"]
    B["Per-connection root broker: restricted signatures, user verification, launch"]
    W["Worker: UID/GID 65534, fresh exec, DTLS and stream parsing"]
    R["Session relay: target account, fresh exec, no host private-key memory"]
    P["Target shell or command: target account and supplementary groups"]
    S -->|"clone3 into private cgroup"| B
    S <-->|"Ciphertext and explicit peer metadata"| W
    B <-->|"Bounded Unix IPC"| W
    B -->|"Set up terminal/pipes, drop privileges, exec"| P
    B -->|"Drop privileges and replace image"| R
    W <-->|"Session IPC after authentication"| R
    R <-->|"PTY or pipes"| P

12.1 Supervisor and worker

The root supervisor loads administrator policy and the host identity, owns UDP listeners, routes ciphertext, records authenticated identities, and manages connection lifetimes. Every admitted connection receives a separate privileged broker. The broker creates a network worker that clears supplementary groups, switches to UID/GID 65534, disables core dumps, sets no_new_privs, closes unnecessary descriptors, and execs a fresh pshd --worker image before processing untrusted protocol bytes.

Exec is a key-memory boundary. Merely freeing an inherited EVP_PKEY cannot guarantee that private bytes are absent from allocator pages inherited by fork. Replacing the address space removes those inherited pages from the worker. The worker’s custom OpenSSL provider contains public Ed25519 material and an IPC signing callback rather than a private host key.

After exec, the worker becomes non-dumpable and installs an x86-64 seccomp filter. It denies process creation and exec, process inspection and signal sending, new sockets and connection setup, namespace changes, and several same-UID interference paths. It also restricts fcntl and resource-limit operations. These restrictions matter because different workers share a service UID.

The filter is a denylist supplement to privilege separation, not a complete filesystem sandbox. Workers still hold live transport secrets and plaintext. UID 65534 is reserved for workers and rejected as a remote account target. Service stdin/stdout are redirected to /dev/null; inherited diagnostic access is write-only so workers cannot read the administrator’s controlling terminal.

12.2 Authentication, launch, and relay exec

Only the privileged broker authorizes the account and starts the process. After valid proofs it performs account checks, creates a PTY or separate pipes, initializes supplementary groups, drops GID/UID, closes privileged descriptors, changes to the account home, and execs its configured shell.

The broker reports the authenticated complete client identity to the supervisor, then drops to the target account and execs a fresh session-relay image. This second exec removes inherited host private-key memory before post-authentication worker traffic reaches the relay. The relay retains process descriptors and its worker IPC socket. Long-lived protocol relaying therefore does not require a root process holding host keys for that connection.

The resulting shell has the ordinary authority of its Unix account. It is not confined by the worker filter. A non-root account that can invoke sudo or a setuid program retains whatever privilege those mechanisms permit. Disallowing direct root login does not establish a system-wide ban on privilege escalation by an authorized account.

12.3 Local IPC

Authentication/session IPC uses Unix SOCK_SEQPACKET; ciphertext routing uses Unix datagrams. Local messages carry native uint32 type and length fields and a bounded payload. Exact received sizes and declared lengths are checked before interpretation. These structures must never be copied directly onto the Internet.

The local registry is:

The socket role and phase determine which operations are legal. IPC is a security interface even though it is local: the worker is an untrusted caller of privileged services. Worker, broker, and relay executables must match.

13. Session semantics and application flow control

13.1 Account and shell policy

Login requires an existing configured account and authorized complete public pair. Accounts with UID 0 or 65534 are rejected. Homes must be absolute, shells must be executable absolute paths, and nologin/false shells are rejected. The launcher checks shadow account expiry, forced password-change state, and password age expiry even though no password authentication occurs. A locked password hash by itself is not treated as account revocation; authorization remains public-key based.

The child begins with umask(077) and a controlled environment:

HOME=<account home>          USER=<account name>
LOGNAME=<account name>       SHELL=<configured shell>
PATH=/usr/local/bin:/usr/bin:/bin
TERM=xterm-256color          LANG=C.UTF-8

Daemon and client environment variables are not forwarded. Shell startup files and subsequently executed programs can of course modify their own environment. The initial TERM value is fixed rather than negotiated.

An empty command starts the configured shell with a login-style argv[0]. A command is passed to that shell with -c. Client command arguments are joined with spaces; psh does not preserve a remote argv vector. Quoting and expansion therefore require the same care as any remote shell command string. This feature is unsuitable as an unescaped wrapper around untrusted command fragments.

13.2 PTYs, pipes, signals, and output

Without a PTY, stdout and stderr travel in separate frames. With a PTY, the terminal combines them. The client defaults to a PTY for an interactive empty command with terminal stdin; -t and -T explicitly select PTY or pipe behavior. The client sets local terminal mode for the session and restores terminal state and descriptor flags during cleanup.

RESIZE carries rows and columns; the broker checks them and updates the PTY. SIGNAL permits only HUP, INT, QUIT, TERM, KILL, USR1, USR2, CONT, STOP, and TSTP. The relay signals the shell’s process group. Wire signal values follow this Linux-targeted implementation rather than a platform-independent SSH signal-name registry.

The relay alternates stdout/stderr priority to avoid permanent starvation of one pipe. EXIT follows output drainage and contains the shell’s exit status; signal termination maps to 128 + signal. The client accepts statuses through 255 and returns them. Transport/setup failure also returns 255, so that numeric result alone cannot distinguish a remote exit 255 from a connection error. CLOSE acknowledges completion or requests early closure.

13.3 Why transport credit is insufficient

A transport receiver can consume incoming bytes while the remote process has stopped reading stdin. Without a second credit system, those bytes can accumulate in session queues or force the relay to block, preventing SIGNAL or CLOSE from being processed. Healthy transport keepalives do not mean a blocked command is consuming input.

READY grants exactly 8,192 bytes of application input credit. The client reserves credit before reading local stdin, so locally read bytes, queued STDIN frames, and broker-pending bytes remain covered by one window. The broker independently subtracts received bytes and returns INPUT_CREDIT only as it writes input to the process, or deliberately discards it after the input side has closed.

The broker queues at most 8,192 input bytes and continues processing IPC while the process pipe is blocked. SIGNAL, CLOSE, and RESIZE do not consume input credit. Empty STDIN, over-credit input, repeated EOF, and STDIN after EOF close the session. The client rejects credit that would raise its available window above 8,192. Receive-window transport probes do not replenish application credit.

This bounds the amount of process input that can stand ahead of a control request. It does not create an independent priority stream or remove transport loss delays.

13.4 EOF and half-close behavior

EOF consumes no credit and is applied only after earlier queued input drains. For pipes it closes the input descriptor. For a PTY it queues one byte 0x04, the conventional VEOF character, after prior input. That synthetic byte does not consume or return client credit. Its effect depends on terminal settings; it is not a universal out-of-band terminal half-close.

When a pipe reader closes stdin, subsequent credited input is discarded and its credit returned while stdout/stderr and EXIT continue. On PTY input hangup the relay similarly discards input but retains the shared master descriptor long enough to drain final output. Closing a shared PTY master immediately would lose output and final status along with input.

14. Liveness, revocation, and complete process cleanup

14.1 Liveness is authenticated peer activity

PING/PONG keepalives run every 15 seconds independently of process progress. The channel expires after 120 seconds without authenticated traffic from the validated peer. Unauthenticated packets and an unvalidated candidate do not extend that peer lifetime. A quiet shell remains live when its peer continues to answer; this is a transport liveness timeout, not a user-inactivity policy.

Connection failure closes the relay path and triggers process cleanup. Ordinary cleanup sends HUP, TERM, and KILL to the process group with bounded waits. The launcher observes child exit using waitid(..., WNOWAIT) and defers reaping until the signaling sequence is finished, reserving the numeric PID/process-group ID against reuse during escalation.

14.2 Supervisor-owned cgroup lifetime

Process-group cleanup alone cannot cover every descendant. A child can change session or process group, and a stopped or crashed relay cannot cooperate. psh therefore creates a private root-owned cgroup v2 directory for each connection. clone3(CLONE_INTO_CGROUP) places the broker inside it atomically, before it can fork workers or shells. All descendants inherit membership.

Only the root supervisor retains the scope directory handle. Workers, relays, and commands inherit no writable cgroup handles; directories remain root-owned. On forced retirement the supervisor writes cgroup.kill, using the scope rather than a worker-supplied or potentially reused shell PID as the final authority. It reaps the broker and verifies the scope is empty before removing it and recycling the connection slot. Normal completion allows a short worker drain window, approximately one second, before final scope cleanup.

The implementation requires a usable cgroup v2 mount, CLONE_INTO_CGROUP, and permission to create scopes and write cgroup.kill under /sys/fs/cgroup. Unsupported or restricted environments reject admissions without a cooperative fallback. The cgroup is a lifecycle mechanism: no per-session memory.max, CPU, or process-count quotas are established by this design. Uninterruptible kernel sleep may delay an empty scope, and daemon shutdown has a bounded wait.

14.3 Reload and revocation

SIGHUP loads and validates the complete replacement configuration, host pair, and authorized public files before changing active policy. Invalid replacements preserve the old policy. Listener changes reject the reload and require restart.

Accepted reloads cancel pending authentications. An active session whose exact client pair is no longer authorized for its account is retired. Changing either host identity component retires all sessions. Unchanged authorized identities can continue. Supervisor-owned scope termination backs these actions independently of relay cooperation.

Logs include lifecycle events and verified complete-pair fingerprints without recording terminal contents or private-key material. This is operational logging, not a durable command-execution audit trail.

15. Credential storage, configuration, and build policy

15.1 Local key protection

Private files must be regular files owned by the expected account, with no group/other permissions; generation uses mode 0600. Host private files are root-owned. Protected public files require the expected owner and no group/world write permission. Loaders reject final-path symlinks, special files, NUL bytes, and files above 64 KiB. They inspect the opened inode and bound actual reads, rather than relying only on an earlier path-based metadata check. Administrators must also keep the containing directories under the appropriate control.

A nonempty local passphrase encrypts both PKCS#8 objects using PBES2, PBKDF2-HMAC-SHA256 with 600,000 iterations, independent fresh 16-byte salts, and AES-256-CBC with separate IVs. Both objects use the same passphrase and must share encrypted/plaintext mode. An empty passphrase or explicit unattended generation produces plaintext private objects. Host keys are plaintext for unattended startup and rely on file permissions and host security.

This passphrase is never a remote login credential. PBKDF2 protects a stored file against password guessing; HKDF serves the unrelated connection key schedule. PBKDF2 is not memory-hard, and the current PKCS#8 CBC encryption has no AEAD tag. The paper therefore does not claim that psh’s at-rest key format is categorically stronger than other private-key formats.

Generation exclusively publishes complete private and public bundles, refuses to overwrite existing paths, and flushes them. These are separate publications; a crash between them can leave a complete private bundle without its public companion. Failed public publication removes only the private file just created. psh-owned private encoding buffers and passphrases are cleansed; OpenSSL owns decoded key objects until freed. The client releases its identity objects after authentication, without claiming that this is equivalent to the server’s exec boundary.

15.2 Strict configuration

Configuration is a deliberately restricted YAML schema, with file paths resolved relative to the configuration file. Server configuration and authorized key files are administrator-managed. Client configuration is owned by the current user. Configuration rejects unsafe ownership or group/world write permission, final-path symlinks, special files, and input beyond 64 KiB.

The parser allows one document and at most 4,096 nodes with depth at most 16. It rejects aliases, anchors, explicit tags, merge keys, duplicate keys, unknown fields, YAML directives, NULs, and excessive scalars. Silent interpretation of an unknown security setting is avoided by rejecting the document.

Schema limits include eight listeners, 32 configured users, 16 authorized public files per user, and 32 client aliases. There is no root-login, password-login, cipher-downgrade, or forwarding switch. Example listener addresses are wildcard IPv4/IPv6 on UDP port 2222; choosing a nonstandard port is not authentication or a security boundary.

15.3 Pinned dependencies and installation

The build uses vendored OpenSSL 4.1.0 and libyaml 0.2.5, with static linkage to the project-built libraries. Release archive hashes and an exact unpacked source inventory are verified. Unexpected files, missing files, symlinks, special files, and checksum mismatches fail verification. Dependency builds do not fetch network content or substitute host crypto/YAML libraries. See the dependency policy and build script .

Unused OpenSSL applications, tests, dynamic modules, legacy providers/protocols, QUIC, and compression are disabled by the build policy. Project-owned C enables compiler warnings, stack protection, PIE, RELRO, and immediate symbol binding. These choices narrow and stabilize build inputs; they do not by themselves establish a reproducible binary build or eliminate the dependency update burden. Static libraries require an explicit rebuild and redeployment when updated.

Installation preserves existing configuration and identities. It creates no keys or login accounts and starts or enables no service. Startup requires an explicitly provisioned host pair and account authorization. During development, all builds, runtime tests, listener tests, and account tests belong in the disposable QEMU guest through tools/vm/vm.sh; host services and networking are not modified.

16. Comparing policy with contemporary OpenSSH

16.1 Versioned comparison

The comparison below is dated 2026-11-11. It concerns upstream behavior; distribution packages and local policy may differ. OpenSSH has preferred hybrid post-quantum agreement since 9.0 and made mlkem768x25519-sha256 the preferred method in 10.0. See OpenSSH’s PQ overview .

OpenSSH 10.6 enables the hybrid ssh-mldsa44-ed25519 signature algorithm, and 10.6p2 adds it to allowed CA signature algorithms. Claims that contemporary OpenSSH has no PQ authentication support are therefore obsolete. These facts are recorded in the 10.6 release announcement and 10.6p2 release notes .

Agreement history and fallback are documented in the PQ overview . Signature support is documented in the 10.6 announcement . Server policy defaults are from sshd_config(5) ; host-checking policy is from ssh_config(5) . The standard SSH transport is described in RFC 4253 .

16.2 What the stricter policy buys

For a deployment that wants only paired-public-key access to non-root accounts, psh makes that policy the available implementation path. There is no password method to disable, no accidental classical-only negotiation to permit, and no forwarding or multi-session channel policy to reconcile with account rules. The broker’s acceptance condition is small enough to state directly: one authorized complete client pair, one bound request, and two valid signatures.

The mandatory profile also makes interoperability failure visible. An old peer, unavailable required primitive, or incomplete identity fails setup. Operators must upgrade both endpoints and provision new complete-pair pins rather than relying on a compatibility negotiation path. This sacrifices reachability in exchange for a uniform accepted-session policy.

Complete-pair provisioning, bounded input credit, supervisor-enforced session lifetimes, and early host trust are useful properties to examine in their own right. Their correctness depends on implementation details; a smaller feature matrix is a reason to expect fewer interactions to review, not a measurement of vulnerability prevalence.

16.3 What the comparison does not establish

OpenSSH already confirms unknown host keys by default; that is shared behavior, not a psh invention. OpenSSH can be configured for public-key-only authentication, no direct root access, and a restricted algorithm policy. Its privilege-separated design also predates this project; contemporary OpenSSH architecture is documented in its privilege-separation description .

The absence of forwarding in psh removes protocol handlers but does not stop an authorized unrestricted shell from running a forwarding program. Likewise, an allowlisted account can run network tools according to its operating-system permissions. The narrow protocol is not a network-egress sandbox.

OpenSSH provides capabilities psh lacks, including mature deployment workflows, certificates and hardware-backed identity options. psh currently provides no PAM or multi-factor path, per-key forced-command policy, or subsystem/file-transfer service. Those omissions can be decisive disadvantages rather than safer choices.

No side-by-side vulnerability-rate study, formal security comparison, or matched OpenSSH performance benchmark has been performed for this paper. psh’s selected defaults are more restrictive for its stated use case; overall superiority to OpenSSH is not an established result.

17. Resource limits as part of the protocol

Bounds apply at different layers and are not interchangeable. A 64 KiB reliable queue is not the entire connection footprint; application buffers, IPC/kernel queues, DTLS allocations, process stacks, and cryptographic objects also exist. Likewise, a 64-connection service cap does not bound the processes an authorized shell can fork. The implementation should be assessed as a system of finite protocol queues and admission limits, not as a demonstrated hard memory quota.

18. Validation evidence and experimental results

18.1 Existing verification

The psh/3 validation record reports full normal tests, sanitizer suites, and adversarial real-client tests inside the disposable QEMU/KVM guest. It records zero compiler warnings on final full build/test/sanitizer runs and two fresh independent agent verification runs. Those are internal project checks and do not constitute an external security audit or peer-reviewed protocol analysis.

The test coverage includes:

The validation record documents a consequential experimental correction: source sync once allowed stale host build artifacts to overlay guest binaries. Affected results were discarded and checks rebuilt and rerun. Root build/ is now excluded from sync. This matters because a passing test is evidence about the binary actually executed, not automatically about the source being reviewed.

Sanitizers cover project-owned C, not the vendored static OpenSSL/libyaml code. Unit runs enable leak detection; credential-dropping integration runs disable LeakSanitizer tracing. The 100,000-iteration deterministic parser mutation harness is a bounded smoke test rather than a long-running coverage-guided campaign. Some properties, including the full 64-connection saturation limit and IPv6 link-local zone behavior, have source-review evidence rather than every desired direct integration experiment.

18.2 Recorded paired-authentication measurements

The existing psh/3 normal run reports 30 sequential sessions and 16 concurrent sessions in the disposable Ubuntu x86-64 guest. The recorded measurement harness uses guest loopback; the PQ validation environment is documented as two vCPUs and 2 GiB RAM. Values below reproduce the psh/3 record without extrapolation:

All observed UDP payloads were at most 1,200 bytes. The harness measures “hello-to-ready” from its first observed DTLS ClientHello to receipt of the command’s PQ_READY output marker. That includes subsequent transport setup, host/user authentication, shell launch, and marker delivery, but excludes the earlier cookie admission exchange. It is not an isolated primitive or DTLS-handshake benchmark. CPU is aggregate over the reported run, not per-session latency. Wire totals exclude IP/UDP headers and include admission, application/control traffic, and closure. See the measurement harness .

RSS sums sampled process resident sets, including shared pages and overlapping closing connections. It is not unique physical memory, a hard memory cap, or a per-connection allocation measurement. These small loopback experiments establish neither wide-area performance nor superiority over OpenSSH. Their narrower result is that the implemented paired profile completed the recorded workloads within the packet ceiling.

18.3 Reproduction boundary

The documented reproduction entry point is:

tools/vm/vm.sh start
tools/vm/vm.sh wait
tools/vm/vm.sh sync
tools/vm/vm.sh run 'make -j2 all && make test && make test-sanitize'
tools/vm/vm.sh stop

Application binaries and privileged/listener/account tests must not run on the host. The helper uses QEMU user networking and copies source without a writable shared host filesystem. Guest setup may download an operating-system image and install guest tools; dependency and application builds use the checked-in inputs offline. Focused instrumented real-client runs have additional commands and log locations in the validation record .

19. Limitations and research directions

The highest-value next work concerns assurance and measurement rather than adding algorithm choices. The following are open directions, not implemented features or deployment guarantees:

1. Formal analysis of paired application authentication. Model the split Ed25519-DTLS/ML-DSA host proof, exporter binding, exact OPEN commitment, and compromised-worker capability. State precisely which authentication claims survive which compromises.
2. External audit and sustained fuzzing. Review privileged IPC, key decoding, source attribution across DTLS transitions, credential drops, and cgroup lifecycle behavior. Extend parser mutation into sustained coverage-guided testing while keeping all runtime work inside the disposable VM.
3. Wide-area transport evaluation. Measure heterogeneous RTT, random and burst loss, reordering, MTU constraints, congested links, and coexistence with other traffic. Compare equivalent cryptographic/authentication policies when using OpenSSH as a performance baseline.
4. Explicit workload quotas. Evaluate administrator-managed cgroup resource policy beyond lifetime termination. Current protocol bounds do not stop an authorized account from consuming its ordinary system resources.
5. Key management at scale. Study hardware-backed paired identities, revocation distribution, and stronger at-rest protection without weakening complete-pair trust or silently adding a remote password path.
6. Cryptographic maintenance. Reassess primitive parameters, pinned-library bounds, and upgrade policy as standards and implementations evolve. A fixed profile reduces negotiation states but makes coordinated migration essential.
7. Recovery semantics. Any future detach/reattach or command-retry facility needs explicit authorization, replay, and execution semantics. Connection migration alone supplies none of those properties.

20. Design assessment

psh/3 implements a coherent narrow remote-shell policy: mandatory hybrid agreement, mandatory paired host and user identities, complete-pair trust before client disclosure, independent broker authorization, and explicit bounds from admission through session I/O. The process architecture treats private-key memory and descendant cleanup as operating-system boundaries, using exec and cgroup lifetime control alongside cryptographic checks.

Its strict defaults reduce policy choices for deployments that accept this scope. Their costs are equally concrete: coordinated upgrades, larger credentials and proofs, Linux-specific lifecycle requirements, a small transport window, and missing authentication and administration features. The recorded tests support implementation claims within their stated environment. Stronger claims about protocol security, operational robustness, or comparative safety require the further evidence identified above.

References and implementation map

External references were consulted on 2026-11-11. Links to rolling documentation may change; OpenSSH release announcements supply the version-specific comparison.

Repository-relative links below identify the implementation to inspect when a behavior or bound changes: