#define private public - Claus Brod on stuff (October 2015)

Macbook hangs after installing OS X 10.11.1 (31 Oct 2015)
"Disk FOO wasn't ejected because one or more programs may be using it!" (25 Oct 2015)

Macbook hangs after installing OS X 10.11.1 (31 Oct 2015)

In more than two years of owning a Macbook, I never faced a single issue upgrading the OS. Until today when, after installing the OS X 10.11.1, the Macbook would just hang after rebooting.

More precisely, the progress bar on the boot screen would stop at roughly two thirds of the way. So today, finally, I had to figure out what kind of diagnosis and recovery options OS X actually provides.

The two most important tools were:

⌘+V during reboot: Boot in verbose mode, showing the boot messages instead of the Apple logo and progress bar. I noticed that the boot process hung after a message saying pci pause: SDXC.
⌘+R during reboot: Boots into recovery mode, which offers tools for disk and networking diagnosis, as well as the option to open a terminal as a superuser.

Things I tried:

Reset the System Management Controller (SMC)
Reset the NVRAM - this actually changed the behavior; the progress bar would now fill up to 100%, but then nothing happened anymore. When booting in verbose mode (showing all the boot messages), startup continued past the pci pause message, but stalled later in what looked like a loop involving messages liked IO80211AWDLPeerManager::setAwdlOperatingMode Setting the AWDL operation mode from SUSPENDED to AUTO and IO80211AWDLPeerManager::setAwdlAutoMode Resuming AWDL. (See also http://superuser.com/questions/917220/macbook-can-not-boot-os-x-yosemite-as-it-hangs for a description of similar symptoms.)
Repair my SSD in recovery mode - this found a number of issues on the disk and repaired them, but did not fix the hang-up issue.
Fix permissions using the recovery mode resetpassword utility
Boot info safe mode - this did not work at all, as the hang-up apparently affected safe mode as well.

I was dangerously close to reinstalling the OS now, with only two aces left up my sleeves: Single-user mode and kernel extensions.

I had seen references to startup issues related to kernel extensions in some articles, but knew very little about them. The first thing I tried was to temporarily disable kernel extension signing (nvram boot-args=kext-dev-mode=1), with no discernible effect. Fortunately, at this point I came across Justin Silver's blog article "OS X El Capitan 10.11.1 Hanging on Boot" in which he describes a startup issue which looks a lot like mine. His excellent summary contains instructions on disabling kernel extensions using recovery mode - which helped me solve my problem!

I booted into recovery mode, opened a terminal window, and then:

  $ mount -rw /
  $ cd /Volumes/Macintosh\ HD
  $ cd Library/Extensions
  $ ls
ACS6x.kext                   HighPointIOP.kext
ATTOCelerityFC8.kext         HighPointRR.kext
ATTOExpressSASHBA2.kext      PromiseSTEX.kext
ATTOExpressSASRAID2.kext     SoftRAID.kext
AX88179_178A.kext            hp_io_enabler_compound.kext
ArcMSR.kext                  hp_io_printerclassdriver_enabler.kext
CalDigitHDProDrv.kext

Following Justin's advice, I moved all kernel extensions out of /Library/Extensions to disable them temporarily, and then rebooted. Lo and behold, the login screen appeared! After a few such rounds of moving kernel extensions back to /Library/Extensions and rebooting, the root cause of my startup issues turned out to be those HP printer extensions.

Phew.

The HP kernel extensions had been on my system for quite some time, of course, and had never caused any problems like this before. So there are still open questions about what exactly had happened here. To be solved on a rainy day.

"Disk FOO wasn't ejected because one or more programs may be using it!" (25 Oct 2015)

I had seen that pesky message every now and then before, but today I finally got fed up with it, as I was pretty sure that no application was running which had a legitimate business touching the external drive - at least none of the applications I had started myself. So I set out on a quest to find the culprit.

The weapon of choice in such a case is lsof:

  sudo lsof | grep /Volumes/FOO

In my case, this produced a list of files opened by a process called mds_store which, apparently, is used to produce the Spotlight search index:

mds_store 42059            root  txt       REG                1,6       3277  8832664 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/live.0.indexGroups
mds_store 42059            root  txt       REG                1,6      32768  8832663 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/live.0.indexIds
mds_store 42059            root  txt       REG                1,6       8192  8832666 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/live.0.indexTermIds
mds_store 42059            root  txt       REG                1,6       8192  8832668 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/live.0.indexPositionTable
mds_store 42059            root  txt       REG                1,6       8224  8832669 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/live.0.indexDirectory
mds_store 42059            root  txt       REG                1,6       1024  8832670 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/live.0.indexCompactDirectory
mds_store 42059            root  txt       REG                1,6      65536  8832671 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/live.0.indexArrays
mds_store 42059            root    5r      DIR                1,6       2074  8832643 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842
mds_store 42059            root   11r      DIR                1,6       2074  8832643 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842
mds_store 42059            root   47u      REG                1,6         28  8832650 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/indexState
mds_store 42059            root   49u      REG                1,6     118784  8832674 
  /Volumes/FOO/.Spotlight-V100/Store-V2/A75D8EF4-8412-4A13-8775-A52C20F05842/.store.db
backupd   42148            root    4w      REG                1,6       1300  8832803
  /Volumes/FOO/Backups.backupdb/Claus/2015-10-25-175619.inProgress/.Backup.467484979.422497.log

The fun part was that I had explicitly configured Spotlight to ignore that particular external disk - and yet, it was still trying to index it!

Turns out that I am not alone with this. "Disable Spotlight on a FAT32 external drive" provides the best summary I could find. Apparently, the type of file system on the external drive plays a role. I was somewhat skeptical about this claim, but then, all my external drives had FAT32 file systems on them, and so I followed the instructions in the article. I was still somewhat incredulous, as this seemed to be such a basic issue and the article is rather old.

But then, following the instructions indeed seemed to be successful, at least initially. The magic ingredient in the sauce was to create a top-level file called .metadata_never_index on the affected drive:

touch /Volumes/FOO/.metadata_never_index

I was too impatient to wait for existing instances of mds_store to finish their work, and did not care about the consistency of the Spotlight index on the external drive anyway. So I killed the mds_store process right away, then unplugged the drive and plugged it in again. From there, indeed I had no issues anymore with ejecting the external disk. Hmmm...