At an enterprise customer, a routine platform update (a categorization engine update) completed without visible issues — services restarted cleanly, no interruption in data flow. But the update left a silent side effect: while regenerating the web server (nginx) configuration files, it also reverted the currently valid SSL certificate back to an old, default one.
The symptom: "connection not secure"
Shortly after the update, visiting the platform in a browser triggered a "your connection is not private" warning. The certificate had expired weeks earlier — a valid, up-to-date certificate was already installed, but the update process had ignored it and regenerated its own default reference.
Root cause: config drift inside a container
The platform ran its web service inside a Docker container, with the SSL configuration mounted from a directory on the host. When the update script regenerated the nginx config, it also overwrote the certificate reference at that mount point, bringing the old, expired certificate file back into play.
This is a classic side effect of automated updates: the update assumes it owns every file it touches, but when part of the config is managed manually/externally, an update can silently reset it to a stale default.
The fix
- Located the container's SSL config path.
- Re-uploaded the already-backed-up, valid certificate files (with months of validity remaining).
- Reloaded the web service and verified the connection.
- Re-confirmed with the customer the next day — no further issues.
The entire fix took minutes, before the customer even noticed.
Takeaways
- "The update completed successfully" doesn't mean side-effect-free. Verifying that critical config (SSL, DNS, firewall rules) is still as expected should be a separate step after every update.
- Config drift is sneakier in container-based services. A file on the host can be unexpectedly rewritten by a process inside a container — keep mount points and ownership of each config piece explicit.
- Backup discipline is what makes "silent" failures fast to fix. Because a current certificate backup was on hand, the fix took minutes, not hours.
- Be cautious with automated updates. Especially in production, running a quick "critical surfaces" checklist (SSL, access, core functionality) right after an update is the cheapest way to catch a problem before the customer does.