whats-the-performance-impact-of-intel-spectre-and-meltdown-within-vmware-environments

Two massive security flaws: Intel Spectre and Meltdown with VMware environments

We want to share this summary very important topic and show some interesting links regarding these two massive security flaws. There are many blog posts and tips going around, here is an abstract.

Intel Spectre and Meltdown

Please scroll down for this important issue and learn how to check your environment and track your remediation!

First of all, here is VMware’s article regarding this issue:

Purpose

VMware is aware of the CPU vulnerabilities that may result in side-channel analysis due to speculative execution, which impacts, amongst other products, VMware vSphere ESXi. Ensuring customer security is our top priority.

VMware has released updates and patches which mitigate known variants of the speculative execution vulnerabilities identified by CVE-2017-5753, CVE-2017-5715 (Spectre), and CVE-2017-5754 (Meltdown). As is our practice, VMware will continue to assess any further security risks, and will continue to provide updates and patches as appropriate.

Customers have inquired if there may be a performance cost associated with either the VMware mitigations, or mitigations of the guest operating systems as released from the OS providers. This knowledge base article will be used as the centralized document for which performance data relating to the speculative execution mitigations published.

This document will focus on Performance data related to Spectre/Meltdown. Please review KB52245: VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) for a holistic view on VMware’s response to these issues.

Resolution

VMware is currently evaluating the performance costs of the Meltdown/Spectre mitigations for VMware products. We plan to test with a wide variety of workloads using both unpatched and patched guest operating systems to provide a comprehensive view of relevant performance characteristics. We will be updating this KB with our data as results become available.

Spectre and Meltdown

Photo courtesy of Andrea Mauro

Rafia Shaikh wrote on Wccf (Where Consumers Come First) tech:

VMware has started to reissue patches and workarounds for its affected Virtual Appliance products that are vulnerable to the Meltdown and Spectre security flaws. The company said its VMware VA products, including vCloud Usage Meter (UM), Identity Manager (vIDM), vCenter Server (vCSA), vSphere Data Protection (VDP), vSphere Integrated Containers (VIC), and vRealize Automation (vRA) are affected.

Publishing its advisory, the firm said that CPU data cache timing can be abused to “leak information out of mis-speculated CPU execution, leading to (at worst) arbitrary virtual memory read vulnerabilities across local security boundaries in various contexts.” If successful, the exploitation can lead to information disclosure.

For VMware products, please check the correct advisories to learn more about the workaround until permanent fixes are made available:

  • vCloud Usage Meter (UM): KB52467
  • Identity Manager (vIDM) 3.x, 2.x: KB52284
  • vCenter Server (vCSA) 6.0, 6.5: KB52312 [5.5 isn’t affected]
  • vSphere Data Protection (VDP): Unavailable
  • vSphere Integrated Containers (VIC): Patch available
  • vRealize Automation (vRA): 7.x KB52377 | 6.x KB52497

For more details, check out this security advisory.

Simon Sharwood wrote on The Register:

Proper patches under way, but for now – to your command lines, vAdmins! Here is an abstract of Simon’s article:

VMware has advised on how to mitigate the Meltdown and Spectre chip design flaws in several of its products.

The workarounds cover vCloud Usage Meter, Identity Manager (vIDM), vCenter Server, vSphere Data Protection, vSphere Integrated Containers and vRealize Automation (vRA). And they’re important because VMware now ships several of its products as appliances: vCenter, for example, is no longer allowed to run in a Windows VM.

The knowledge base articles for all the products state that Meltdown and Spectre can create problems for virtual appliances, explain that the mitigation tactics will stop attacks but must be considered "a temporary solution only and permanent fixes will be released as soon as they are available."

Andrea Mauro posted on vInfrastructure Blog:

Meltdown and Spectre are critical vulnerabilities existing in several modern CPU: these hardware bugs allow programs to steal data which is currently processed on the computer. Meltdown and Spectre can affect personal computers, mobile devices, server and several cloud services.

To check if you need the patches in your vSphere environment see also this post: Meltdown and Spectre: check a vSphere environment.

Solution

  • VMware vSphere 6.5: for ESXi apply patches ESXi650-201712101-SG (released on Dec, 19th 2017), ESXi650-201801401-BG, ESXi650-201801402-BG; for vCenter Server (and PSC) upgrade to version 6.5 U1e
  • VMware vSphere 6.0: for ESXi apply patches ESXi600-201711101-SG and ESXi600-201801401-BG, ESXi600-201801402-BG; for vCenter Server (and PSC) upgrade to version  6.0 U3d
  • VMware vSphere 5.5: for ESXi apply patches ESXi550-201709101-SG (this patch has remediation against CVE-2017-5715 but not against CVE-2017-5753) and ESXi550-201801401-BG; for vCenter Server (and PSC) upgrade to version 5.5 U3g
  • VMware Workstation 14: update to version 14.1.1
  • VMware Workstation 12.x: update to version 12.5.9
  • VMware Fusion 10: update to version 10.1.1
  • VMware Fusion 8: update to version 8.5.10

The previous version of vSphere (like 5.0 or 5.1) are no more supported, so one reason more to upgrade fast your infrastructure to a supported version!

Important note from the new VMware KB 52345 in case you have installed (or you are planning to install) VMware’s initial microcode patches ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG. Check first your processor model and family!

Intel has notified VMware of recent sightings that may affect some of the initial microcode patches that provide the speculative execution control mechanism for a number of Intel Haswell and Broadwell processors. The issue can occur when the speculative execution control is actually used within a virtual machine by a patched OS. At this point, it has been recommended that VMware remove exposure of the speculative-execution mechanism to virtual machines on ESXi hosts using the affected Intel processors until Intel provides new microcode at a later date. Check the VMware KB 52345 to verify the affected CPU.

Also, ensure that your VMs are using Hardware Version 9 or higher (this is mandatory). For best performance, Hardware Version 11 or higher is recommended. VMware Knowledge Base article 1010675 discusses Hardware Versions.

Anyway, it’s still not enough. There are other requirements, if applicable:

  • Deploy the Guest OS patches for CVE-2017-5715. These patches are to be obtained from your OS vendor.
  • Update the CPU microcode. Additional microcode is needed for your CPU to be able to expose the new MSRs that are used by the patched Guest OS. This microcode should be available from your hardware platform vendor.

For all VMs you have to power off (a guest reboot or VM reset it’s not enough!) in order to activate the hypervisor assisted protection!

Note that actually there isn’t yet a specific patch for the other several virtual appliances from VMware (like vSphere Replication or NSX Manager), potentially there is no need of specific patches… but anyway the best way is stay tuned for future comments or news. In part the new VMware KB 52264 reply to this question.

Note that all appliances affected, must be patched, that powered off (a VM reboot it’s not enough) and then powered on again.

Please read here Andrea’s complete article, there is a lot more information!

Also Ivo Beerens published a very helpful article on his blog IVOBEERENS.nl

Currently these security flaws can be divided into the following categories:

Spectre and Meltdown

Photo courtesy of Ivo Beerens

Operating System patches will protect against number variant 1 and 3.  With variant 2 a CPU microcode update is required.

Ivo showed some examples and many other helpful links. Please see here his complete article.

And also from Ivo:

More information:

Spectre and Meltdown

Photo courtesy of Ivo Beerens

Thanks a lot to the whole community for all the information and your great work!

We also want to do our job in supporting you as a customer and integrated Spectre/Meltdown dashboard that can be seen as a mixture between a checkup tool and a dashboard-guided check-list.

After you selected the VMware vCenter, Datacentor or Cluster to get information from, the dashboard provides you with the current status of your environment. 

  1. How many ESXi Hosts have the correct microcode or not
  2. How many have VMs are patched
  3. How many VMs have the correct virtual hardware version

As we can already guess that there will be a negative performance impact based on the microcode and other Spectre/Meltdown patches, there is currently no real proof or statistical validation.

Therefore, it is important to compare the performance of your past reference period to Now, after you patched the systems.

That is exactly what we do! We let you know what performance impact just happened because of the patches. A crucial information to your infrastructure planning!

Intel Spectre and Meltdown Check-List

Check it out by yourself and test your environment using Performance Analyzer  – Free 30 day Trial!!

Download Performance Analyzer

CNIL
Metrics and Logs

(formerly, Opvizor Performance Analyzer)

VMware vSphere & Cloud
PERFORMANCE MONITORING, LOG ANALYSIS, LICENSE COMPLIANCE!

Monitor and Analyze Performance and Log files:
Performance monitoring for your systems and applications with log analysis (tamperproof using immudb) and license compliance (RedHat, Oracle, SAP and more) in one virtual appliance!

Subscribe to Our Newsletter

Get the latest product updates, company news, and special offers delivered right to your inbox.

Subscribe to our newsletter

Use Case - Tamper-resistant Clinical Trials

Goal:

Blockchain PoCs were unsuccessful due to complexity and lack of developers.

Still the goal of data immutability as well as client verification is a crucial. Furthermore, the system needs to be easy to use and operate (allowing backup, maintenance windows aso.).

Implementation:

immudb is running in different datacenters across the globe. All clinical trial information is stored in immudb either as transactions or the pdf documents as a whole.

Having that single source of truth with versioned, timestamped, and cryptographically verifiable records, enables a whole new way of transparency and trust.

Use Case - Finance

Goal:

Store the source data, the decision and the rule base for financial support from governments timestamped, verifiable.

A very important functionality is the ability to compare the historic decision (based on the past rulebase) with the rulebase at a different date. Fully cryptographic verifiable Time Travel queries are required to be able to achieve that comparison.

Implementation:

While the source data, rulebase and the documented decision are stored in verifiable Blobs in immudb, the transaction is stored using the relational layer of immudb.

That allows the use of immudb’s time travel capabilities to retrieve verified historic data and recalculate with the most recent rulebase.

Use Case - eCommerce and NFT marketplace

Goal:

No matter if it’s an eCommerce platform or NFT marketplace, the goals are similar:

  • High amount of transactions (potentially millions a second)
  • Ability to read and write multiple records within one transaction
  • prevent overwrite or updates on transactions
  • comply with regulations (PCI, GDPR, …)


Implementation:

immudb is typically scaled out using Hyperscaler (i. e. AWS, Google Cloud, Microsoft Azure) distributed across the Globe. Auditors are also distributed to track the verification proof over time. Additionally, the shop or marketplace applications store immudb cryptographic state information. That high level of integrity and tamper-evidence while maintaining a very high transaction speed is key for companies to chose immudb.

Use Case - IoT Sensor Data

Goal:

IoT sensor data received by devices collecting environment data needs to be stored locally in a cryptographically verifiable manner until the data is transferred to a central datacenter. The data integrity needs to be verifiable at any given point in time and while in transit.

Implementation:

immudb runs embedded on the IoT device itself and is consistently audited by external probes. The data transfer to audit is minimal and works even with minimum bandwidth and unreliable connections.

Whenever the IoT devices are connected to a high bandwidth, the data transfer happens to a data center (large immudb deployment) and the source and destination date integrity is fully verified.

Use Case - DevOps Evidence

Goal:

CI/CD and application build logs need to be stored auditable and tamper-evident.
A very high Performance is required as the system should not slow down any build process.
Scalability is key as billions of artifacts are expected within the next years.
Next to a possibility of integrity validation, data needs to be retrievable by pipeline job id or digital asset checksum.

Implementation:

As part of the CI/CD audit functionality, data is stored within immudb using the Key/Value functionality. Key is either the CI/CD job id (i. e. Jenkins or GitLab) or the checksum of the resulting build or container image.

White Paper — Registration

We will also send you the research paper
via email.

CodeNotary — Webinar

White Paper — Registration

Please let us know where we can send the whitepaper on CodeNotary Trusted Software Supply Chain. 

Become a partner

Start Your Trial

Please enter contact information to receive an email with the virtual appliance download instructions.

Start Free Trial

Please enter contact information to receive an email with the free trial details.