Date of incident - 2019-05-14
Time event began - 2:13am PDT
Project services restored - 6:15AM PDT
Final resolution - Clean up the incursion and upgrade “Script Security” Plugin for Jenkins to avoid repeat infection.
Root cause - shell escape exploit in the “Script Security” plugin for Jenkins, remotely injected via a specially crafted URL. The exploit caused the Jenkins server to crash, but succeeded in running the payload. A bitcoin miner was installed under the Jenkins user for FD.io Jenkins.
Other contributing causes:
Downtime for projects: FD.IO Jenkins
Minor downtime/interruptions for other project Jenkins systems as they were updated.
2:13am PDT: exploit is successfully performed on FD.io Jenkins via a specially crafted URL containing the payload:
107.174.x.x - - [14/May/2019:09:13:26 +0000] "GET /descriptorByName/org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript/checkScript?sandbox=True&value=public+class+x%7Bpublic+x%28%29%7Bnew+String%28%22[payload_omitted]%22.decodeHex%28%29%29.execute%28%29%7D%7D HTTP/1.1" 200 6 "-" "python-requests/2.21.0"
The exploit caused Jenkins to crash, but the payload execution was successful due to critical vulnerabilities in the “Script Security” Jenkins plugin. The payload downloaded and executed the following shell script:
The script is not Jenkins-specific, so it does not attempt to modify any content owned by Jenkins. Since the script runs as a non-privileged user, most actions fail without any effect. The script does install a cryptocoin miner (“cryptonight”) and puts itself into crontab for the jenkins user in order to persist across reboots.
2:19am PDT: Monitoring recognizes that Jenkins is down and issues an alert, which is received and acknowledged by C.Hoy Poy (sysops on call).
2:48am PDT: C.Hoy Poy identifies that there has been a security incursion and shuts down the system per security first responder procedure.
2:56am PDT: The issue is escalated to K.Ryabitsev (Director of IT Security), J.Conway (SysOps Team Lead).
4:00am PDT: The incursion is traced down to the Jenkins “Script Security” plugin and payload is identified and analyzed. The team takes a decision to clean up the affected system instead of reinstalling it, since there is no indication that the payload succeeded to do anything beyond installing the cryptocoin miner.
6:15am PDT: The system is cleaned up and all plugins are updated to their latest security errata versions. Jenkins is brought back online and all services restored.~2:00pm PDT: All other Jenkins systems are analyzed for traces of the same incursion and upgraded to the latest security errata. 2019-05-14 fd.io Jenkins security incident