Fixes are being rolled out.
Cisco vEdge SD-WAN devices subject to bricking after expired certificates are reloaded onto the device.
Cisco released an informational post that “describes how to identify a vEdge that has an expired certificate affecting control plane connections, which eventually impacts data plane connections resulting in loss of service.” The expired certificate affects the vEdge 1000, vEdge 2000, and vEdge 100M/B platforms and could result in loss of service if improperly handled or if not handled fast enough. Cisco specifically tells users experiencing loss of connection to not reload their device as this could lead to a complete loss of service. Cisco writes, “Reloading the device causes the Graceful Restart Timers to reset and the router will not be able to reconnect to the fabric. Keeping the router up will help ensure Graceful Restart does not occur, which will help to keep the DataPlane (BFD) Sessions up and traffic will be able to pass while control connections are down.” However, simply not restarting your device might not be enough to stave off loss of service, The Register explained, “And bear in mind, even if you don't manually restart or update your equipment, there are timers in the devices that will, by default, start a reload that will trigger disruption as a result of the now-dead cert.”
Cisco is working to fix the problem.
Cisco has begun rolling out software updates. The company has so far released twelve software patches to various versions of the vEdge software. The Register writes, “Based on the documentation, the patch likely amounts to certificate replacement. Unfortunately it doesn’t appear that the update will do much good for devices that have already been rendered inoperable by the expired certs. Cisco recommends customers with bricked gateways contact Cisco for assistance.” Cisco has also released step by step processes to remedy the issue and correctly install the update along with remarks for customers who have reloaded their devices prior to reading the post. Cisco gave instructions on remedying a device that has been restarted and is now unresponsive. Cisco explained that it will “require an OOB connection or a physical visit” to the device and change the system time to before the certificate expires.