FD.io Jenkins Incident


Vanessa Valderrama
 

Description: The Jenkins executors were not spinning up as expected.

Cause: A vendor SSL was changed at the service endpoint and because the
TTL was not updated Jenkins had the old DNS cache.

Solution: We cleared the LF cache and restarted Jenkins which restored
service.

Follow-up: We are working with the vendor to improve advanced
communication regarding these types of changes in the future to prevent
unplanned downtime.

Join tsc@lists.fd.io to automatically receive all group messages.