Green Cloud Technologies
Investigating - As most of you are likely aware, Intel has had a bad year with bugs and vulnerabilities associated with their CPU feature 'Speculative Execution.' On August 14th, they announced yet another vulnerability in their processors known as "Foreshadow / Foreshadow NG / Spectre Variant 4 / L1 Terminal Fault" (CVE-2018-3615/3620/3646). RedHat has created an excellent video explaining this vulnerability: https://www.youtube.com/watch?v=kBOsVt0iXE4

As with previous variants, this new vulnerability has possible significant performance impacts when the available patches are applied. These patches currently disable Hyper-Threading on the processor, and regularly clear the L1 cache of each core, so we need to carefully evaluate the possible impact to performance and capacity if they are to be applied. Disabling Hyper-Threading effectively cuts our core count by one third, which would require significant capacity augments to avoid impacts to our customers - which we will implement, but not until neccesary. We are also working closely with VMware to determine if a version of their patch can avoid disabling Hyper-Threading by using a new scheduler called Core Scheduling, much like Microsoft's HyperClear. It is important to note: if and when this option becomes available, it will only be effective when most or all of the guest OSs in our infrastructure are patched as indicated below.

In addition to the possible performance impact, this variant requires multi-layer mitigation. We must patch all of our blades' CPU microcode, the hypervisor code on those blades, and the guest OS must be updated with patches from the OS vendor. For Microsoft, the OS patches are available now through security updates in the form of Security Rollup, Security Only, or Security Updates (see the specific Microsoft link below for more information). Again, full remediation will require that these OS patches are in place at the time that Green Cloud patches its systems.

If you are concerned about any non-Green Cloud systems, we highly recommend the videos below, and that you take the appropriate actions detailed in the vendor links below.

As always, security at Green Cloud is of the highest priority, but mitigation in this case is not a simple reactive task. We are researching, testing, and working with our vendors to ensure we have the best solution, always mindful of the goal to implement the updates as quickly as possible. We plan to move quickly from research and analysis to implementation immediately following our final planning session, scheduled for Monday, August 20th. More specific information on actions and the timeline will be provided at that time.

We have provided links to various vendor pages on the subject of the vulnerabilities below. There you can find more details on the vulnerabilities and additional information regarding the OS patches, which should be applied as soon as possible.

RedHat Video: https://www.youtube.com/watch?v=kBOsVt0iXE4
Intel Video: https://www.youtube.com/watch?v=n_pa2AisRUs

https://foreshadowattack.eu/
https://en.wikipedia.org/wiki/Foreshadow_(security_vulnerability)
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3615
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3620
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3646
https://www.kb.cert.org/vuls/id/982149

https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00161.html
https://software.intel.com/security-software-guidance/software-guidance/l1-terminal-fault
https://support.hpe.com/hpsc/doc/public/display?docLocale=en_US&docId=emr_na-hpesbhf03874en_us
https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180814-cpusidechannel
http://support.lenovo.com/us/en/solutions/LEN-24163
https://www.synology.com/support/security/Synology_SA_18_45
http://www.huawei.com/en/psirt/security-advisories/huawei-sa-20180815-01-cpu-en
https://security.netapp.com/advisory/ntap-20180815-0001/
http://xenbits.xen.org/xsa/advisory-273.html
https://support.f5.com/csp/article/K95275140
https://psirt.global.sonicwall.com/vuln-detail/SNWLID-2018-0009
http://www.vmware.com/security/advisories/VMSA-2018-0021.html
https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180018
Aug 17, 15:50 EDT

About This Site

Welcome to Green Cloud's system status site. Please subscribe to updates above to get up to minute alert notifications on any scheduled maintenance activities and/or service interruptions. If you require further assistance, please email us at support@gogreencloud.com or call us at 1.877.465.1217

Network Operational
Network - Greenville, SC   ? Operational
Network - Nashville, TN   ? Operational
Network - Houston, TX   ? Operational
Network - Atlanta, GA   ? Operational
Network - Phoenix, AZ   ? Operational
Network - Minneapolis, MN   ? Operational
IaaS ? Operational
IaaS - GSP1   ? Operational
IaaS - NVL1   ? Operational
IaaS - GSP2   ? Operational
IaaS - Houston   ? Operational
IaaS - Atlanta   ? Operational
IaaS - Phoenix   ? Operational
IaaS - Minneapolis   ? Operational
DRaaS ? Operational
DRaaS with Zerto - Greenville, SC   ? Operational
DRaaS with Zerto - Nashville, TN   ? Operational
DRaaS with Zerto - Atlanta, GA   ? Operational
DRaaS with Zerto - Phoenix, AZ   ? Operational
DRaaS with StorageCraft - Greenville, SC   ? Operational
DRaaS with StorageCraft - Nashville, TN   ? Operational
DRaaS with StorageCraft - Atlanta, GA   ? Operational
DRaaS with StorageCraft - Phoenix, AZ   ? Operational
DaaS ? Operational
DaaS - Greenville, SC   ? Operational
DaaS - Nashville, TN   ? Operational
DaaS - Atlanta, GA   ? Operational
DaaS - Phoenix, AZ   ? Operational
BaaS ? Operational
BaaS with Veeam - Greenville, SC   ? Operational
BaaS with Veeam - Nashville, TN   ? Operational
BaaS with Veeam - Atlanta, GA   ? Operational
BaaS with Veeam - Phoenix, AZ   ? Operational
Security   ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
GSP1 vCloud ?
Fetching
GSP2 vCloud ?
Fetching
NVL1 vCloud ?
Fetching
HOU vCloud
Fetching
ATL vCloud
Fetching
PHX vCloud
Fetching
Past Incidents
Aug 20, 2018

No incidents reported today.

Aug 19, 2018

No incidents reported.

Aug 18, 2018

No incidents reported.

Aug 16, 2018

No incidents reported.

Aug 15, 2018

No incidents reported.

Aug 14, 2018

No incidents reported.

Aug 13, 2018

No incidents reported.

Aug 12, 2018

No incidents reported.

Aug 11, 2018

No incidents reported.

Aug 10, 2018

No incidents reported.

Aug 9, 2018
Resolved - This incident has been resolved.
Aug 9, 08:14 EDT
Monitoring - Due to a failure of a hypervisor compute node in our Nashville datacenter, a small number of virtual servers and virtual firewalls/routers were automatically restarted onto other compute nodes. This restart activity concluded by 5:32 Eastern Time. Please contact Green Cloud Support if you have any questions or issues regarding this incident.
Aug 9, 06:11 EDT
Aug 8, 2018
Completed - The scheduled maintenance has been completed.
Aug 8, 21:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 8, 19:00 EDT
Scheduled - Green Cloud will be performing an urgent maintenance at 7pm eastern on 8/8/2018 that will cause a temporary loss of management control of your environments via vCloud and the Green Cloud portal. Our goal is to minimize the impact as much as possible so as to reduce any effect on you or your customers and there is no expected impact on the actual operation of your virtual machines or networks. This is purely a maintenance on management components. Please contact Green Cloud Support if you have any questions or issues regarding this maintenance.
Aug 8, 14:00 EDT
Resolved - The issue has been resolved. The Network Operations Center will continue to monitor the infrastructure for stability and will close the issue when we feel the steps taken have cleared all symptoms. We strive to proactively identify any lingering issues resulting from this issue, but please contact us immediately if at this time your service is not performing up to your expectations. We can be reached at 877-465-1217 or support@gogreencloud.com. Again, we sincerely apologize for the inconvenience this issue is causing both you and your customers.
Aug 8, 11:36 EDT
Monitoring - Due to a failure of a hypervisor compute node in our Nashville data center, a small number of virtual servers and virtual firewalls/routers were automatically restarted onto other compute nodes. This restart activity concluded by 10:45 am Eastern Time. Please contact Green Cloud Support if you have any questions or issues regarding this incident. 877-236-4159 or Support@gogreencloud.com
Aug 8, 10:48 EDT
Identified - Due to a failure of a hypervisor compute node in our Nashville datacenter, a small number of virtual servers and virtual firewalls/routers were automatically restarted onto other compute nodes. Please contact Green Cloud Support if you have any questions or issues regarding this incident.
Aug 8, 10:34 EDT
Aug 7, 2018
Completed - The scheduled maintenance has been completed.
Aug 7, 01:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 7, 00:00 EDT
Scheduled - Green Cloud will be performing potentially intrusive maintenance on a hypervisor compute node in our Nashville datacenter that has experienced a management agent failure. We plan to perform this activity during our normal maintenance window starting at 8-6-2018 at midnight. This maintenance may include a reboot of the host, which would cause virtual machines to automatically restart on other hosts in the cluster although we hope to avoid that if possible. Please contact Green Cloud Support if you have any questions or issues regarding this maintenance.
Aug 6, 16:12 EDT
Aug 6, 2018

No incidents reported.