SECURITY ALERT: Latest Intel CPU Vulnerabilities
Incident Report for Green Cloud Defense
Resolved
This incident has been resolved.
Posted Sep 19, 2018 - 11:33 EDT
Update
We are continuing to investigate this issue.
Posted Sep 19, 2018 - 11:32 EDT
Investigating
As most of you are likely aware, Intel has had a bad year with bugs and vulnerabilities associated with their CPU feature 'Speculative Execution.' On August 14th, they announced yet another vulnerability in their processors known as "Foreshadow / Foreshadow NG / Spectre Variant 4 / L1 Terminal Fault" (CVE-2018-3615/3620/3646). RedHat has created an excellent video explaining this vulnerability: https://www.youtube.com/watch?v=kBOsVt0iXE4

As with previous variants, this new vulnerability has possible significant performance impacts when the available patches are applied. These patches currently disable Hyper-Threading on the processor, and regularly clear the L1 cache of each core, so we need to carefully evaluate the possible impact to performance and capacity if they are to be applied. Disabling Hyper-Threading effectively cuts our core count by one third, which would require significant capacity augments to avoid impacts to our customers - which we will implement, but not until neccesary. We are also working closely with VMware to determine if a version of their patch can avoid disabling Hyper-Threading by using a new scheduler called Core Scheduling, much like Microsoft's HyperClear. It is important to note: if and when this option becomes available, it will only be effective when most or all of the guest OSs in our infrastructure are patched as indicated below.

In addition to the possible performance impact, this variant requires multi-layer mitigation. We must patch all of our blades' CPU microcode, the hypervisor code on those blades, and the guest OS must be updated with patches from the OS vendor. For Microsoft, the OS patches are available now through security updates in the form of Security Rollup, Security Only, or Security Updates (see the specific Microsoft link below for more information). Again, full remediation will require that these OS patches are in place at the time that Green Cloud patches its systems.

If you are concerned about any non-Green Cloud systems, we highly recommend the videos below, and that you take the appropriate actions detailed in the vendor links below.

As always, security at Green Cloud is of the highest priority, but mitigation in this case is not a simple reactive task. We are researching, testing, and working with our vendors to ensure we have the best solution, always mindful of the goal to implement the updates as quickly as possible. We plan to move quickly from research and analysis to implementation immediately following our final planning session, scheduled for Monday, August 20th. More specific information on actions and the timeline will be provided at that time.

We have provided links to various vendor pages on the subject of the vulnerabilities below. There you can find more details on the vulnerabilities and additional information regarding the OS patches, which should be applied as soon as possible.

RedHat Video: https://www.youtube.com/watch?v=kBOsVt0iXE4
Intel Video: https://www.youtube.com/watch?v=n_pa2AisRUs

https://foreshadowattack.eu/
https://en.wikipedia.org/wiki/Foreshadow_(security_vulnerability)
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3615
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3620
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3646
https://www.kb.cert.org/vuls/id/982149

https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00161.html
https://software.intel.com/security-software-guidance/software-guidance/l1-terminal-fault
https://support.hpe.com/hpsc/doc/public/display?docLocale=en_US&docId=emr_na-hpesbhf03874en_us
https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180814-cpusidechannel
http://support.lenovo.com/us/en/solutions/LEN-24163
https://www.synology.com/support/security/Synology_SA_18_45
http://www.huawei.com/en/psirt/security-advisories/huawei-sa-20180815-01-cpu-en
https://security.netapp.com/advisory/ntap-20180815-0001/
http://xenbits.xen.org/xsa/advisory-273.html
https://support.f5.com/csp/article/K95275140
https://psirt.global.sonicwall.com/vuln-detail/SNWLID-2018-0009
http://www.vmware.com/security/advisories/VMSA-2018-0021.html
https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180018
Posted Aug 17, 2018 - 15:50 EDT
This incident affected: IaaS (IaaS - Nashville, TN, IaaS - Greenville, SC, IaaS - Houston, TX, IaaS - Atlanta, GA, IaaS - Phoenix, AZ) and Security.