Certificate expiry outages are a design failure, not bad luck
A certificate that expires in production was not forgotten by an individual. It was issued by a process that did not include renewal, into a system that had no way to reload it.
The outage is a symptom of the issuance process
The post-incident review of a certificate expiry usually concludes that a renewal was missed and a reminder should be added. That explanation is accurate about the proximate event and wrong about the cause, which is why the same organisation has the same outage two years later.
The actual cause is that the certificate was issued by a process that ended at issuance. Someone needed a certificate, obtained one, installed it, and the work was complete. Nothing in that sequence created an obligation to renew, a record that the certificate existed, a mechanism to replace it, or a signal when it approached expiry. The renewal was not forgotten; it was never part of the design.
This is why calendar reminders fail. A reminder depends on the certificate being known, on the owner still being in the role, and on the replacement being straightforward when the day arrives. In a large estate, the certificates that cause outages are precisely the ones that fail all three conditions—an appliance configured during a migration, a service-to-service certificate issued by a team that has reorganised, a client certificate in a partner integration.
Four capabilities that prevent it
Preventing the class requires four things, and the absence of any one of them reinstates the problem:
- Inventory. A list of every certificate in the estate, discovered rather than declared, with its subject, issuer, expiry, and the system that presents it.
- Automated issuance and renewal. A certificate is requested, obtained and installed by a process, not a person. ACME provides the protocol for this, and it works for internal authorities as well as public ones.
- Reload without restart. The serving system picks up the new certificate without a deployment or an outage window, because a renewal that requires a change freeze will be deferred.
- Independent expiry monitoring. Something outside the renewal path observes what is actually being presented and alerts well before expiry.
The fourth is the one most often missing where automation exists, and it is the one that catches the failure mode automation introduces: the renewal client that has been silently failing for six weeks because its credential expired.
Reload without restart is the forgotten half
Automating issuance without solving reload produces a system that obtains certificates it cannot use. The renewal succeeds, the new file is written, and the running process continues to present the old certificate until someone restarts it—which happens, if at all, at a moment unrelated to the expiry.
The mechanisms differ by platform and the requirement is the same: the serving component must re-read its certificate material on a signal, on a file change, or on a scheduled interval, without dropping connections. Where a component genuinely cannot do this, the certificate belongs in a terminating proxy that can, rather than in the component.
Two related gaps are worth checking at the same time. Certificates pinned inside application code or configuration—a trusted CA bundle compiled into a client, a fingerprint in a mobile application—cannot be rotated without a release, which makes the release cycle the rotation cycle. And certificates held by counterparties in a mutual TLS arrangement require a coordinated change, which means the design needs an overlap period where both the old and new certificate are accepted.
Internal certificates are the harder problem
Public certificates have improved substantially. Automated issuance is standard, lifetimes are short enough that automation is mandatory, and certificate transparency means an organisation can discover publicly-trusted certificates issued for its own domains, including ones it did not know about.
Internal PKI has none of these properties by default. A private authority typically issues long-lived certificates through a request process, without transparency logging, to systems that were built at different times by different teams. There is often no inventory at all, and the authority’s own certificate—the one whose expiry takes down everything at once—may have been issued a decade ago by someone who has left.
Three actions reduce this materially. First, discover rather than declare: scan the estate and collect what is actually presented, including on internal ports and by scheduled jobs. Second, issue internal certificates only through the same automated mechanism, and shorten their lifetimes so that automation is exercised continuously rather than annually. Third, treat the intermediate and root lifetimes as a planned event with a documented rotation sequence, because they are the certificates whose expiry is a company-wide incident and whose renewal has never been rehearsed.
NIST SP 800-57 frames certificates as one case of keys with defined cryptoperiods and transition requirements, which is the right way to think about the hierarchy: the root’s cryptoperiod is a decision someone made, and its consequences arrive on a specific date.
Shortening lifetimes forces the fix
The instinct when renewals are painful is to ask for longer-lived certificates. It is understandable and it makes the underlying problem permanent.
A long lifetime removes the pressure that keeps automation working. A ninety-day certificate renews roughly every sixty days, so a broken renewal path is discovered within weeks by a process that is supposed to be routine. A three-year certificate renews once, by whoever happens to be on call, using a procedure nobody has run.
The industry has moved decisively in this direction. The CA/Browser Forum baseline requirements have repeatedly reduced maximum validity for publicly-trusted server certificates, and the trajectory continues downward. Organisations that treated automation as optional have found the decision made for them.
Applying the same logic internally is the highest-value change available. It is also the least popular, because it converts a problem that is currently invisible into daily operational activity—which is exactly the point.
Revocation does not work the way people assume
A related misconception is that a compromised certificate can simply be revoked. In practice, revocation checking is inconsistent: clients may not check, checks may fail open to avoid breaking connections, and internal clients frequently have no configured revocation source at all.
The practical consequence is that a certificate’s validity period is close to the real window of exposure for its key. A three-year internal certificate whose private key is compromised in month two is, for most clients, valid for thirty-four more months. This is the second argument for short lifetimes, and it is independent of the operational one.
Where revocation must be reliable—client certificates granting access, code signing, device identity—the design needs a checking mechanism the relying party actually consults, with defined behaviour when the check is unavailable. Deciding that behaviour is a security decision: failing open preserves availability and removes the control; failing closed preserves the control and creates an availability dependency on the revocation service.
What to monitor and from where
Expiry monitoring should observe what clients see, not what the configuration says. Connect to each endpoint, read the presented chain, and alert on the earliest expiry in it—including intermediates, which expire too and are usually not monitored.
Three thresholds work better than one: an informational signal at thirty days, a warning at fourteen, and a page at seven. The gaps let a broken automatic renewal surface as a warning rather than an outage, and the page is the last resort rather than the mechanism.
Monitor the renewal path itself as well. A renewal client that has not successfully obtained a certificate in its expected interval is a failure now, not in three weeks. And monitor the things that are not endpoints: client certificates used by scheduled jobs, certificates inside appliance configurations, and the signing certificates whose expiry breaks validation rather than connection—which fail in ways that TLS handshake monitoring will never see.