Magnus Andersson mentioned in his blog a pretty important VMware bug including the corresponding KB article.

When an APD event occurs, LUNs connected to ESXi may remain inaccessible after paths to the LUNs recover. The 140-second APD timeout expires even though paths to storage have recovered. You see the following events in sequence in the /var/log/vmkernel.log:

  1. Device enters APD.
  2. Device exits APD.
  3. Heartbeat recovery and filesystem operations on the device fail due to "timeout" or "not found" or "busy".
  4. The APD timeout expires despite the fact that the device exited APD previously.

This is of course a real issue in production environments as the LUN doesn’t recover after being gone for some seconds, no matter if the LUN is back online within the 140-seconds timeout.

As the LUN is not accessible anymore all of your virtual machines are affected (down) and some hosts might not come back to be managed in vCenter. Of course the hosts will continue to operate but it happens quite often that they get disconnected in vCenter.

To resolve the issue, you need to kill the outstanding I/O to the LUN and reboot the ESXi Host!

Kill the outstanding I/O:

LUN colllapse APD Timeout

Photo courtesy of sanbeiji(CC ShareALike)

Metrics and Logs

(formerly, Opvizor Performance Analyzer)

VMware vSphere & Cloud

Monitor and Analyze Performance and Log files:
Performance monitoring for your systems and applications with log analysis (tamperproof using immudb) and license compliance (RedHat, Oracle, SAP and more) in one virtual appliance!

Subscribe to Our Newsletter

Get the latest product updates, company news, and special offers delivered right to your inbox.
Share on twitter
Share on linkedin
Share on facebook
Share on email

Subscribe to our newsletter

White Paper — Registration

We will send you the research paper via email.

CodeNotary — Webinar

White Paper — Registration

Please let us know who you are, so we can send you the CodeNotary Trusted Software Supply Chain white paper.

Become a partner

Start Your Trial

Please enter contact information to receive an email with the virtual appliance download instructions.

Start Free Trial

Please enter contact information to receive an email with the free trial details.