When should you design a security protocol, and when should you not?
Almost every custom security protocol should have been a profile of an existing one. The few that are justified share a specific set of constraints - and a review process that assumes the first design is wrong.
The cost you inherit with a new protocol
A standard security protocol arrives with assets that took years to accumulate: published analysis by people who attack protocols for a living, formal models, interoperable implementations, fuzzing corpora, test vectors, deployment guidance, a downgrade history that has already been learned from, and libraries whose flaws have been found by everyone else’s incidents.
A new protocol has none of this. It has your team’s analysis, your team’s implementation, and your team’s test cases—which is to say it has been reviewed by the people least able to see its blind spots.
That asymmetry is the argument, and it is not about competence. TLS 1.3 went through years of academic analysis and multiple formal verification efforts, and the process still found issues in intermediate drafts. A protocol designed in a sprint and reviewed by two engineers is not comparable, however good those engineers are.
Most requirements are satisfied by a profile
Before designing anything, establish whether the requirement is actually a configuration of something existing. The great majority are. A useful mapping:
| Requirement | Existing answer |
|---|---|
| Authenticated, confidential channel between services | Mutual TLS with a defined cipher suite and certificate policy |
| Same over an unreliable datagram link | DTLS 1.3 |
| Encrypt a payload to a recipient’s public key, no session | HPKE |
| Prove a request came from a specific client, end to end through proxies | HTTP message signatures |
| Delegated authorisation between services | OAuth 2.0 with sender-constrained tokens |
| Firmware or artefact authenticity | A signature scheme plus a defined manifest format |
| Device identity attestation | An attestation format over an existing channel |
A profile is real work: choosing versions and suites, specifying certificate validation, defining what happens on failure, pinning what must not be negotiated. That work is worth doing carefully and is far cheaper than a design.
The test for whether you have left profile territory is whether your own messages provide a security property. If your format carries application data inside a TLS session, you have a profile. If your format establishes keys, proves identity or provides replay protection, you have designed a protocol.
The constraints that genuinely force a design
A small set of constraints does defeat the standard options, and they share a shape: a hard physical or regulatory bound that no existing construction fits.
- Message size or round-trip budgets that no handshake meets—a few dozen bytes per message on a constrained radio link, or a single round trip mandated by a real-time control loop.
- Energy budgets where a full handshake is a measurable fraction of battery life over the product’s service period.
- Legacy peers that cannot be changed, where the protocol has to interoperate with equipment that predates modern options and cannot be updated.
- A domain specification you do not control that mandates a particular exchange, common in payments, automotive and industrial control.
- A security property no standard protocol provides—a threshold signing ceremony, a multi-party computation, a specific unlinkability requirement.
Note what is absent from that list: performance concerns that have not been measured, a preference for a simpler implementation, unfamiliarity with the standard option, and a belief that the system is not exposed enough to matter. Each of these appears regularly as a justification and none of them survives contact with the cost of getting a protocol wrong.
State the adversary before the messages
When a design is justified, the first artefact is not a message sequence. It is a written statement of what the adversary can do, because every subsequent decision depends on it.
Specify at least: whether the adversary can read messages, modify them, drop them, reorder them, replay them, or inject their own; whether they can act as a legitimate peer; whether they hold a valid credential for a different peer; whether they can compromise one endpoint and, if so, what they learn; whether they can observe timing and message sizes; and what happens to past sessions if a long-term key is later compromised.
That last question—forward secrecy—is the one most often omitted, and it determines the handshake shape rather than being a property you add later. So does the decision about whether identity must be hidden from a passive observer, which is why TLS 1.3 encrypts the certificate and its predecessors did not.
Write the properties you are claiming as a short list, with the adversary each holds against. A protocol that claims “confidentiality” without saying against whom cannot be reviewed.
The properties a reviewer will ask about
An independent review of a protocol design tends to converge on the same questions, and a team can usefully ask them of itself first:
- Key separation. Does every distinct use have a distinct key derived with a labelled derivation, or is one key used for encryption, MAC and derivation?
- Transcript binding. Does the final authentication cover the entire handshake, including negotiated parameters, so that a modified negotiation is detected?
- Downgrade resistance. Can an active attacker force the weakest mutually supported option, and would the peers notice?
- Replay. What makes each message single-use—a nonce, a counter, a timestamp window—and what happens at the boundary of that mechanism?
- Identity misbinding. Can a message intended for one peer be accepted by another? Is the peer identity included in the derivation?
- Failure behaviour. Does every error path terminate the session, or does some failure leave a partially established state?
- State machine completeness. Is there a defined response to every message in every state, including unexpected ones?
The negotiation questions deserve particular attention because they are where deployed protocols have failed repeatedly. RFC 8996 exists because supporting old versions for compatibility turned out to be a standing vulnerability rather than a courtesy—a lesson worth applying to a new design’s flexibility from the start.
Implementation is where good designs fail
A correct specification implemented incorrectly provides nothing. The recurring implementation defects are well known and worth testing for explicitly: a signature or MAC compared with a non-constant-time function; a certificate chain validated for signature but not for name, expiry or purpose; an error path that logs and continues; a length field trusted before it is bounds-checked; a nonce reused after a reconnection; a random value drawn from a non-cryptographic source.
Two practices catch most of this. The first is a negative test suite: malformed, truncated, reordered, replayed and out-of-state messages, each with an asserted rejection. The second is differential testing against a second implementation—which a custom protocol will not have unless someone builds one, and the cost of building it is part of the cost of the design.
Deprecation is part of the design
A protocol that ships without a version negotiation and a deprecation story has committed its worst version to permanence. Include from the beginning: a version indicator that the peer must honour, a policy for the minimum acceptable version, and a mechanism to refuse an older one once support ends.
The standard protocols provide a useful model here, including the uncomfortable part. Deprecating a widely deployed version takes years of overlap and ends with a deliberate break. Designing for that break at the start—a mandatory version field, no implicit fallback, a configuration that can raise the floor—is much easier than retrofitting it into a protocol that assumed its first version would be its only one.