Green Cloud Technologies
Update - In our continuing effort to keep our partners informed, we'd like to provide you with a quick update. Intel has released updated information on the previously mentioned issue with their CPU microcode updates ( https://newsroom.intel.com/news/root-cause-of-reboot-issue-identified-updated-guidance-for-customers-and-partners/ ). As the page indicates, there is a fix being tested with a general release imminent. We will bring this update in house for testing as soon as it is made available to us. Upon positive confirmation from our testing as well as from intel and industry peers we will roll this update out to our platforms as quickly as is prudent. Industry peers and press have indicated the microcode updates when operational can have adverse effects on the stability of some applications. For this reason OS and application patching continues to be critical to the success of this overall remediation strategy. Thank you for your continued patience as the industry deals with this very complex and wide reaching issue. Green Cloud knows this is of great concern to you and your customers. We are working with your best interest in mind to manage the many variables involved in planning and executing the right plan.

As always, please contact us at 877-465-1217 or send an email to support@gogreencloud.com if there is any support we can provide during this remediation. We will provide another update when we reach the next step in our testing
Jan 22, 16:14 EST
Update - We have slowed plans to deploy the VMware and Intel Microcode patches to our platforms. As noted in our previous update Intel and VMware have discovered and are working to resolve issues with specific Intel Haswell and Broadwell family chips that can lead to platform reboots when the Spectre protections are engaged ( https://kb.vmware.com/s/article/52345 and https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/ ).

"The [Reboot] issue can occur when the speculative execution control is actually used within a virtual machine by a patched OS. At this point, it has been recommended that VMware remove exposure of the speculative-execution mechanism to virtual machines on ESXi hosts using the affected Intel processors until Intel provides new microcode at a later date."

These processor families are deployed widely throughout our environments so the impact to stability could be significant. This creates a new exploit vector in that the original Spectre exploits could be used to cause denial of service reboots to our infrastructure. Also, although the issues sighted have been found in those specific families there has of yet been no indication that other families of CPUs are not also vulnerable to this bug. We feel the risk to our stability is too high in comparison to the risk of exploitation.

With or without this firmware patch the OS level protections installed during Microsoft and Linux patching are required to protect your system from any exploitation. These software protections do incur a performance penalty, primarily in IO intensive situations. The microcode patch should provide relief to this impact when a stable alternative is released by Intel. Also when the microcode update is re-released the OS level patches will still be required to enable the functionality at the hardware level. For this reason we continue to advocate aggressive OS level patching by our partners. This is the best path to protect your customers.

At this time the actions we can take with our hypervisor layer will only provide protections for an as yet un-exploited vulnerability that could be used on an un-patched guest OS to see data within another guest OS. We are currently holding off on this patching as it also introduces another layer of performance impact, and given the impact at the OS level already present we are working to balance performance and risk on your behalf.

We are in close communication with Cisco, VMware and Intel on this issue, and will take immediate action on your behalf as options become available. We know this is a stressful environment for you, your customers and Green Cloud as a whole. We are taking this issue very seriously, and are evaluating our best course of action daily. We will provide more updates on the state of these issues and any actions we take as soon as there is information to share. Thank you for your patience and your continued support.

Again, please contact us at 877-465-1217 or send an email to support@gogreencloud.com if there is any support we can provide during this remediation.
Jan 17, 12:21 EST
Monitoring - Green Cloud continues to work aggressively on your behalf on these issues. Extensive testing has been conducted with our vendor partners as it relates to the impact of the various firmware, hypervisor, and operating system patches required to fully remediate the issue. We intend to begin patching our firmware and hypervisor software in our data centers no later than Monday, January 15th. This patching will be conducted in one data center at a time beginning with our Phoenix, Arizona data center. We will let these patches "soak" in production in this site for 24-48 hours and proceed throughout the network during the week. As always, we will continue to provide you with ongoing updates as they occur.

As stated in the previous update, it is imperative that you continue to patch your operating systems and virus scanning software parallel to our infrastructure patching. You will find information on our performance testing and information related to OS patching below.

Please contact us if you have any questions or concerns related to these vulnerabilities. We understand the pressure placed on your staff by this additional workload and are here to help however we can. Please contact us at 877-465-1217 or send an email to support@gogreencloud.com if there is any support we can provide during this remediation.

What we know at this point:

Our testing of the initial hypervisor patches provided by VMware along with the OS level patches provided by Microsoft, and the Linux community, have shown surprisingly high performance impacts. These range from 10-25% CPU performance degradation. This impact has been a core reason we have delayed the rollout of our hypervisor patches since they account for a 5-10% performance hit on their own. The OS level performance hit is unavoidable until firmware updates are applied. We felt it best to balance the risk of exploit (which remains low) with the impact to performance. Intel has released firmware and microcode updates which we have also tested in addition to the above patches. These updates appear to solve much of the performance impact by providing protections against the exploit (Spectre) in hardware vs limiting capabilities at the OS level to avoid it. Unfortunately, there have been reports (https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/) of this microcode update leading to stability problems with the CPU and causing unpredictable reboots of the compute platform. We continue to monitor the situation with Intel and our other vendors to balance mitigation of the issue with performance and reliability of our platforms. We are hoping we can deploy these firmware patches along with the hypervisor patches next week, but will not do so unless Intel provides an update on the stability.

As it relates to the required Microsoft OS updates, it is important to note the order of operations required for automatic updates to apply. In an attempt to mitigate the BSOD possible when virus scanning software conflicts with the new OS patches, Microsoft has added a registry key that must be toggled by the virus scanner vendor before an automatic application of the patch can occur. Because of this it is imperative that you upgrade your virus scanning software to a supported version and that the vendor toggle this registry key before you can assume automatic updates are occurring. See https://support.microsoft.com/en-us/help/4072699/january-3-2018-windows-security-updates-and-antivirus-software for more details. You can also review https://docs.google.com/spreadsheets/d/184wcDt9I9TUNFFbsAVLpzAtckQxYiuirADzf3cL42FQ/htmlview to see the status of your vendor updates and any caveats related to the registry key.

Again, please contact us at 877-465-1217 or send an email to support@gogreencloud.com if there is any support we can provide during this remediation.
Jan 12, 17:22 EST

About This Site

Welcome to Green Cloud's system status site. Please subscribe to updates above to get up to minute alert notifications on any scheduled maintenance activities and/or service interruptions. If you require further assistance, please email us at support@gogreencloud.com or call us at 1.877.465.1217

Voice   ? Operational
Network Operational
Network - Greenville, SC   ? Operational
Network - Nashville, TN   ? Operational
Network - Houston, TX   ? Operational
Network - Atlanta, GA   ? Operational
Network - Phoenix, AZ   ? Operational
IaaS ? Operational
IaaS - GSP1   ? Operational
IaaS - NVL1   ? Operational
IaaS - GSP2   ? Operational
IaaS - Houston   ? Operational
IaaS - Atlanta   ? Operational
IaaS - Phoenix   ? Operational
DRaaS ? Operational
DRaaS with Zerto - Greenville, SC   ? Operational
DRaaS with Zerto - Nashville, TN   ? Operational
DRaaS with Zerto - Atlanta, GA   ? Operational
DRaaS with Zerto - Phoenix, AZ   ? Operational
DRaaS with StorageCraft - Greenville, SC   ? Operational
DRaaS with StorageCraft - Nashville, TN   ? Operational
DRaaS with StorageCraft - Atlanta, GA   ? Operational
DRaaS with StorageCraft - Phoenix, AZ   ? Operational
DaaS ? Operational
DaaS - Greenville, SC   ? Operational
DaaS - Nashville, TN   ? Operational
DaaS - Atlanta, GA   ? Operational
DaaS - Phoenix, AZ   ? Operational
BaaS ? Operational
BaaS with Veeam - Greenville, SC   ? Operational
BaaS with Veeam - Nashville, TN   ? Operational
BaaS with Veeam - Atlanta, GA   ? Operational
BaaS with Veeam - Phoenix, AZ   ? Operational
Security   ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
vCloud Director Upgrades Feb 22, 00:00-03:00 EST
Green Cloud will be conducting software upgrades to the vCloud Director interface during this window. During this time, the vCloud Director interface will be unavailable for consumption. No virtual server downtime should be experienced during this window. If you have any questions, don't hesitate to respond to this email or call us at 877.465.1217, option 1.
Posted on Feb 19, 09:37 EST
System Metrics Month Week Day
GSP1 vCloud ?
Fetching
GSP2 vCloud ?
Fetching
NVL1 vCloud ?
Fetching
HOU vCloud
Fetching
ATL vCloud
Fetching
PHX vCloud
Fetching
Past Incidents
Feb 19, 2018

No incidents reported today.

Feb 18, 2018

No incidents reported.

Feb 17, 2018

No incidents reported.

Feb 16, 2018

No incidents reported.

Feb 15, 2018

No incidents reported.

Feb 14, 2018

No incidents reported.

Feb 13, 2018

No incidents reported.

Feb 12, 2018

No incidents reported.

Feb 11, 2018

No incidents reported.

Feb 10, 2018

No incidents reported.

Feb 9, 2018

No incidents reported.

Feb 8, 2018

No incidents reported.

Feb 7, 2018
Resolved - At this time, access to vCloud Director has been fully restored. We've isolated the root issue that caused the problem to recur and do not believe it will happen again. Please contact us immediately if at this time your service is not performing up to your expectations. We can be reached at 877-465-1217 or support@gogreencloud.com. Again, we sincerely apologize for the inconvenience this issue may have caused both you and your customers.
Feb 7, 21:09 EST
Update - Green Cloud is continuing to work with our vendor partners to resolve the issues with vCloud Director in Houston. If you need anything while we work to resolve this issue, please call us at 877-465-1217 or send an email to support@gogreencloud.com.
Feb 7, 19:30 EST
Investigating - Green Cloud is investigating a repeated issue with the vCloud Director administration interface in our Houston datacenter and is working to restore access. If you need anything while we work to resolve this issue, please call us at 877-465-1217 or send an email to support@gogreencloud.com.
Feb 7, 17:29 EST
Resolved - This incident has been resolved.
Feb 7, 19:29 EST
Monitoring - On January 29th, 2018 and updated on February 5th, 2018 Cisco released a critical security advisory related to the ASAv line of products. In summary, a vulnerability in the XML parser of Cisco Adaptive Security Appliance (ASA) Software could allow an unauthenticated, remote attacker to cause a reload of the affected system or to remotely execute code. It was also possible that the ASA could stop processing incoming Virtual Private Network (VPN) authentication requests due to a low memory condition affected system or to execute code remotely.

To read the Security Advisory in its entirety, please follow the link to Cisco's website. https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180129-asa1

On February 5th, 2018 Cisco released an update to the ASAv software; version 9.7.1.21 is now available. Green Cloud is suggesting that you upgrade your Partner Managed ASAv as soon as you can. The following links will give you a step by step procedure for the upgrade process. The process should not exceed a 30-minute window.
ASAv upgrade via the Command line interface - http://gogreencloud.com/knowledge-base/asav-upgrade-cli/
ASAv upgrade via ASDM- http://gogreencloud.com/knowledge-base/asav-upgrade-asdm/

For the partners who have Managed Firewall Services form Green Cloud, we will be opening a ticket with you to schedule a maintenance window for this upgrade within the next few business days.

If you need assistance during your upgrade, as always, support is here to help. Please call 877-465-1217 or e-mail support@gogreenclooud.com.
Feb 7, 15:49 EST
Resolved - At this time, access to vCloud Director has been fully restored. Please contact us immediately if at this time your service is not performing up to your expectations. We can be reached at 877-465-1217 or support@gogreencloud.com. Again, we sincerely apologize for the inconvenience this issue may have caused both you and your customers.
Feb 7, 15:50 EST
Investigating - Green Cloud is investigating an issue with the vCloud Director administration interface in our Houston datacenter and is working to restore access. If you need anything while we work to resolve this issue, please call us at 877-465-1217 or send an email to support@gogreencloud.com.
Feb 7, 14:22 EST
Feb 6, 2018
Completed - The scheduled maintenance has been completed.
Feb 6, 02:55 EST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 6, 01:00 EST
Scheduled - Green Cloud will be conducting infrastructure upgrades in our Nashville, TN datacenter. Customers may incur a brief disruption to our BaaS and DRaaS services in that datacenter as well as access to the vCloud Director management web interface during this window. If you have any questions, don't hesitate to respond to this email or call us at 877.465.1217, option 1.
Feb 2, 09:28 EST
Feb 5, 2018

No incidents reported.