Solved

Meltdown & Spectre Vulnerabilities

  • 4 January 2018
  • 17 replies
  • 10139 views

Badge +3
Anyone knows if Meltdown and/or Spectre Vulnerabilities are impacting Nutanix Infastructure?

https://spectreattack.com/

Thank you,
icon

Best answer by smzksts 5 January 2018, 13:19

Dear Alexander,

Please check the Security Advisory #0007 in support portal.
https://portal.nutanix.com/#/page/static/securityAdvisories (Login required)
View original

17 replies

Badge +2
Dear Nutanix support,

please provide an impact assessment, security advisory and patches at your earliest convenience.

Other vendors do this in a timely manner and provide public statements, ideally via US-CERT who handle coordination of this vulnerability:

https://www.kb.cert.org/vuls/id/584653

Yours, sincerely

Alexander List
Badge +5
Dear Alexander,

Please check the Security Advisory #0007 in support portal.
https://portal.nutanix.com/#/page/static/securityAdvisories (Login required)
Badge +3
Thank you for updating us on this matter.
Userlevel 7
Badge +30
No problem dstjean.

We published security advisory #7 on Jan 4th about this.

As a side note, if you've got a portal.nutanix.com account (which all customers can sign up for), you can get email updates for all field and security advisories by logging into portal.nutanix.com, clicking on your name in the top right hand corner, then preferences.

Here's what my preferences look like (see screenshot below). Click on email for everything you're comfortable with, and you'll get them as they are posted

Badge
It should based on the underlying OS and the CPUs used in most deployments.
Userlevel 7
Badge +30
There are multiple attack vectors, just in general, and yes its highly to do with both OS and CPU.

We go into solid detail in that security advisory and are actively digging in here.
Badge
But that's not public, is it? How can I, as a potential customer, assess the way you respond to new vulnerabilities like this if you hide the information from non-customers. I can see when patches land for IBM, Dell, HP, Cisco UCS, Azure, AWS, Google Cloud, but not Nutanix. Please reconsider your approach, there is no reason to keep patches secret!
Badge
As a potential customer I get no information about the Nutanix platform. Why is this hidden in a non-public space??
Userlevel 7
Badge +30
Thanks for reaching out, good point paulw_wwf. Certainly nothing intentionally secret here, its simply the way our site is structured.

In the interim of making that content searchable, as a potential customer, you're welcome to post on our public forums (which is here) and we're happy to collaborate in front of the whole world, no worries.

Or, if you're in content with an account manager, systems engineer, or reseller/partner, you can always ask them and they can route content as appropriate.

Anyhow, I've pinged out security team to inquire why that part of the portal is login only, if there is a specific reason, etc
Userlevel 7
Badge +30
quick follow up paulw_wwf - Talked to security team. They're aware of this ask (others have asked too), and its on the plan to get it done.

I think, besides this, you'll find Nutanix to be an incredibly transparent company. Anything you want, unless its NDA'd, we'll give you freely
Badge
Thanks for the update. If it's just down to the site structure, perhaps you could put a snapshot of the current vulnerability and patch status in this thread. I'm particulary interested in the Intel CPU microcode updates which are required to mitigate Spectre. In general, these are delivered via a BIOS update. Our HP laptops got a BIOS update with the patches in mid December, our CIsco UCS Blades are due an update on 18th Feb. Are there planned BIOS updates for Nutanix hardware, and if so, what are the timelines?

Thanks for listening,

Paul W.
Userlevel 7
Badge +30
The microcode updates depend on a mix of hypervisor and hardware. Some hypervisors (namely ESX and AHV) can load the new microcode as a side load upon boot, after being upgraded to the appropriate version.

Others, like Hyper-V, can not yet do that, so you need to strictly depend on a BIOS Update.

The BIOS update, in general, is going to be a good idea, and we're wrapping a few other goodies in there that we've been working on. We're working on that, date hasn't been set. Tenative was ~Feburary.

That said, you may have seen some manufacturers pulled their new BIOS updates (Dell did that with 13G yesterday) and Intel issued an advisory *yesterday* saying they are seeing some reboot issues in the field:
https://newsroom.intel.com/news/intel-security-issue-update-addressing-reboot-issues/

We're taking the time to really make sure we get this right, as industry-wide these patches were incredibly rushed because of the early embargo break throwing the extra week of engineering time into a frenzy.

The latest copy of the Spectre/Meltdown advisory is available in PDF form here: http://download.nutanix.com/alerts/Security-Advisory_07-v5.pdf

You'll see the exact comment in our advisory as:
---The availability of BIOS versions with stable CPU microcode updates for NX models is under evaluation

Anyhow, for your request, here's a screenshot of all advisories that are posted as of today:

Badge
Thanks Jon. I was aware Linux could push new CPU microcode, I was not aware that ESXi and AHV were doing the same. That certainly makes the BIOS updates much less important, so I appreciate the heads up. And I do wish to congratulate you on the excellent information provided in the PDF, it's a very good overview of the situation. As an engineer I like it.

I'm not sure the data you're publishing actually helps me much with my conversations with senior managers. These tend to be "Are we protected?" "Only partlally, it's complicated" "OK, so when will we be fully protected?". They would much prefer some target dates for either patches, or confirmation that updates aren't required. Things like "under evaluation" or "when it's ready" don't sit well with them.

Just to clarify, are you saying that the latest AHV and ESXi patches do contain fixed microcode, making BIOS updates academic? Or just that they could do? Again, thanks for listening, and thanks for all the technical updates you've provided.
Userlevel 7
Badge +30
RE management conversations
Understandable. You'll find that Nutanix as a company is maniacal about security, and the system is already hardened by default. AOS itself is a closed system, where you can't run 3rd party code. That doesn't remove every attack vector under the sun, but it means we do have a wee bit of time to get it right, rather than rush a patch to our core storage system.


To be clear, we are NOT taking the approach that other vendors have taken (cough cough, ryhmes with "net-lap" cough cough), where they state thing like:
"Unlike a general-purpose operating system, does not provide mechanisms for non-administrative users to run third-party code. Due to this behavior, is not affected by either the Spectre or Meltdown attacks." That's not fingerpointing, its a fact, those net-lappers did that in their public response.

In our mind, that's the "easy way out", and we don't think that's the right way to treat our customer's systems.

Even though that same statement is true for AOS, we're still evaluating steps to harden the AOS against this issue. I can't comment on the specifics because the patches aren't done yet, but just know that we're taking the extra time to get this right. We're not planning on punting this like those other guys.

RE Microcode
Yes, AHV and ESXi patches contain fixed microcode.

VMware will confirm the same here: https://kb.vmware.com/s/article/52085 - See point three under the resolution. Basically, apply the BIOS update OR apply the ESXi patch.

We're still planning on releasing update BIOS either way, but just know that for AHV and ESXi, you get coverage in software to begin with.
Userlevel 7
Badge +30
FYI Version 6 of the update here: http://download.nutanix.com/alerts/SecurityAdvisory07-v6.pdf
Badge +4
Jon has you covered on the tecnical front, but I wanted to jump in and thank you for your feedback, and others on the thread as well for the same. As the one that typically writes the Security Advisories I wanted to thank you for the kinds words, this one took a while to write and it was quite the team effort.

That being said, your feedback on the "senior Manager conversation" is a great one. It's a tough balance. Finding that half way point between enough information to feed the Engineer while trying to avoid it becoming a tech paper. I'll put some thought to that, see how we can better deliver the message so it's useful in more conversations. Thank you for taking the time to provide your thoughts. They are very valuable and most appreciated.
Badge +4
Security Advisory 7, update 7 has been posted to the portal.

http://download.nutanix.com/alerts/Security-Advisory_07_v7.pdf

Reply