Sunday, August 28, 2011

The WinSxS disaster in Windows 8/7/Vista due to Component Based Servicing

In case you don't know, the Windows operating system's servicing stack underwent a complete rewrite in Windows Vista. The primary reason Windows Vista and Windows 7 are so bloated and so horribly slow at installing updates is because of the servicing mechanism they use - you know, the one which is used to install updates and patches. Let me give some background for those who haven't heard about this issue at all.

Windows XP (and Windows 2000) used a fast and great mechanism called Hotfix Installer (Update.exe) to install updates. Updates installed in very little time. If you wanted to further reduce update times on Windows XP, you could just temporarily stop the System Restore service and updates would install at crazy speeds. Note that this is not recommended for novice users who don't know advanced recovery methods, as some updates can sometimes cause your system to stop booting so you cannot even uninstall them. The method the Hotfix Installer used was simple, it just installed a new version of files to be updated at %windir%\system32 and %windir%\system32\dllcache (the Windows File Protection cache). For files that were in use, a restart copied them from dllcache to the system32 folder. This is simple file-based servicing. The hotfix installer (Update.exe) also supported various command line switches like /nobackup which means not to backup files it patches. Again, this is not recommended for novice users as some updates can screw your system even after the comprehensive testing Microsoft does before releasing them. But if you won't be uninstalling any updates (usually one only requires uninstalling updates if they cause problems), you could save a ton of disk space by not backing up the files it patched. The Hotfix Installer backed up files to C:\Windows\$Uninstall$KBxxxxxx folders so even if you did back up the files at install time, they could be safely deleted after a few days if no stability issues were found after using Windows with the newest updates applied. Update.exe also supported the very important and convenient ability to slipstream a service pack or update into the original Windows setup files using the /s switch.    

When Microsoft was developing Windows Vista, they realized that components had gotten too many interdepencies on each other and to service each file reliably without breaking another component that relied on it, Microsoft introduced what they called as Component Based Servicing (CBS). You can read all about it in a much more technical way at The Servicing Guy's blog. What CBS does basically is it installs all files of the entire operating system, including all languages into C:\Windows\WinSxS and then it hard-links files from there to C:\Windows\system32. This has the benefit of not having to insert the OS disc to add or remove any components, and some other advantages as well like offline servicing of a Windows Vista or Windows 7 image. But the design introduces a major disadvantage of taking up a lot of hard disk space. Whenever an update is installed, it no longer installs it to C:\Windows\system32 and C:\Windows\system32\dllcache like Windows XP's hotfix installer (Update.exe) did. Instead, it updates the files in C:\Windows\WinSxS. Now, Windows keeps multiple copies of the same file but with different version in WinSxS if it is used by more than one Windows component. The higher the number of components, that many number of times the file exists in C:\Windows\WinSxS. When a Windows Vista update (.MSU) is installed, the components get updated, each and every one, instead of the files and the worst part is it still maintains the older superseded previous versions of components in WinSxS so the user would be able to uninstall updates.
Microsoft does say that some sort of "scavenging" or deleting older copies of components takes place but is scarce on the details. The scavenging seems to take place automatically at certain intervals in Windows 7 but not in Windows Vista. In Windows Vista, you have to add or remove any Windows component for the scavenging to take place. And Microsoft says the scavenging will free up some disk space but in practice, on my system, I see my free disk space only decreasing on Vista as I remove or add any component. Windows does not give the user an option to not backup the earlier versions of components like Windows XP's /nobackup switch in Hotfix Installer did. As as you install more and more updates on your system, they will take more and more disk space. This is one of the primary reasons Windows Vista and Windows 7 are so bloated. Another reason for them being so bloated is the DriverStore that these OSes store. All drivers that are shipped with the OS and the OEM ones which you download and which are installed for a particular system are staged in C:\Windows\System32\DriverStore. But let's not go there for now.

Now, an important thing to note is that the size of the WinSxS folder is not what Explorer or the dir command report, it is far less but is misreported by Explorer because it counts the hard links more than once when calculating size. That does not mean, the size of WinSxS is not causing real-world disk space problems on numerous Windows Vista/7 systems in use today. Microsoft's ingenious recommendation to this problem of ever growing disk consumption is to install fewer updates to keep the size of the servicing store under control. Of course, users cannot deny installing security updates and leave their system open to security holes. What they can do is install less optional updates, the ones that Microsoft releases on the fourth Tuesday of every month and also install less of the hotfixes that are available by request from a Knowledge Base article. In short, you have to trade the number of bugs fixed in the OS by installing hotfixes at the cost of enormous amounts of disk space. The whole servicing stack is a total downgrade to Windows XP's update.exe method. It causes heavy disk thrashing and slow logoffs/logons while Windows configures these updates at the Welcome Screen. Many systems are unable to boot because of failed updates. Another disadvantage of the "new" servicing stack (and the redesigned Setup mechanism of Windows Vista) is the inability to do a true slipstream of service packs and hotfixes.

The time it takes to actually install these hotfixes online compared to Windows XP is also completely unacceptable. When you start installing an MSU update, it spends a lot of time determining whether the update applies to your system. Then, the update itself takes much longer to install compared to Windows XP's Update.exe (hours instead of minutes if you are installing dozens of updates through a script). Finally, that post-installation process ("Configuring updates... Do not turn off your computer") takes several minutes before shut down followed by a second post-installation process (configuration) upon restart before logon that also takes also several minutes and thrashes the disk.
I can install the entire SP3 for Windows XP in about 10 minutes after downloading the full installer. I can also install a slipstreamed-with-SP3 copy of Windows XP is about 45 minutes on a modern fast PC. In contrast, Windows Vista or Windows 7 do install relatively quickly (in just about 15-20 minutes) on a modern PC but installing the service packs and updates takes more time than anything on XP did. Not only can service packs not be slipstreamed, but Vista Service Packs are not even cumulative, which means if you clean install Windows Vista today, you have to install SP1 first which takes about 90 minutes, then SP2 which takes less time, then all the post-SP2 updates which do take hours to install. If you really HAVE to use Windows 7 or Windows Vista, you are stuck with this slow update non-sense as Microsoft does not even acknowledge that there is any slowdown or loss of functionality in the new servicing mechanism. The fact remains: MSU updates are slow as **** and take too much time and as Windows 7/Vista get older and Microsoft stops producing service packs, a clean install is going to take longer and longer to bring it up-to-date with all patches installed. Is is worth wasting your time on an OS whose servicing mechanism Microsoft completely screwed up? I once again recommend you read more about the servicing stack and how it operates at The Servicing Guy's blog: To fix this messed up servicing stack, Microsoft also offers a tool called CheckSUR for your system if it finds “inconsistencies in the servicing store”.
Microsoft's Windows Vista and Windows 7 products are not engineered with disk space in mind. It causes a problem, especially for SSDs which are still low capacity and very expensive. The only hope is that Microsoft again completely redesigns this servicing mechanism in a future Windows release so it would not cause this growing disk space consumption issue, speed up installation of updates by an order of magnitude, not slow down logon and logoff, not prevent systems becoming unusable because of failed updates being stuck at a particular stage and allow true slipstreaming.

Microsoft's response to this is vague - they simply state "Windows 7's servicing is more reliable than Windows XP" but they cannot acknowledge it is a million times slower and still unreliable...slow to the point of being unusable and sometimes leaving systems in an unbootable damaged state. Of course they know all this too but can't admit it since it makes their latest OSes look poor. Moving from a very simple and fast update mechanism that worked to a complex one that requires endless “configuring” and repair through CheckSUR is a product engineering defect.

Take a look at servicing-related complaints in Microsoft's own forums:

1. Very slow install of updates to Windows 7
2. Windows 7 - Updates are very slow
3. Windows 7 Ultimate, it takes long time configuring updates
4. "Preparing To Configure Windows. Please Do Not Turn Off Your Computer"
5. Very slow update install at shutdown (Windows 7 Home Premium)
6. Why does my computer run so slow when installing updates?
7. Every time the computer is shut down, it always says installing update do not turn off your computer
8. Computer is working slow and wants to do windows updates all the time
9. Windows 7 Update install time taking a very long time
10. Windows wants to install 6 updates every time I log off or put the computer in sleep mode
11. Problem In Configuring Windows Updates at the time of Startup
12. Computer really slow after latest updates
13. Windows hangs up in "configuring updates"
14. Why can't windows 7 install updates?
15. Every time computer is shut down, receive Installing updates, do not shut off....
16. How long does it take for the Windows 7 Home Premium updates take?
17. Windows 7 "Installing Update 2 of 2" for 12 hours now
18. Updates causes endless reboots
19. Updates stuck installing for over 24 hrs. Computer does not boot
20. Cannot load Windows 7 after installing 2 critical updates

A proper solution to this problem would be to completely re-engineer and rewrite the servicing mechanism so it operates with the speed, reliability and pain-free operation of the XP servicing mechanism.

Update: I don't see this situation improving in Windows 8 either. In Windows 8, Microsoft has made some worthwhile improvements to the servicing stack. One is that you can completely remove some component packages (also called payloads) and free up some disk space using the dism /online /remove-package command. Second improvement is that there is finally built-in support for repairing corruption of the servicing store, which means you no longer require downloading CheckSUR. Dism has been modified to let you clean up the CBS store in various ways. Still, it takes a ridiculous amount of time to do the cleanup. It's no resolution to the extremely slow performance of installing updates and of superseded hotfixes continuing to bloat up your hard drive or SSD by default. Your average end user isn't going to go and run DISM. The complexity of the whole thing and learning curve for a command line tool like DISM.EXE is too much for him. So they have added a "Windows Update Cleanup" handler to "Disk Cleanup" but it takes an insane amount of time! Realizing how badly designed the whole thing is, they have backported some of the cleanup features to Windows 7 as well but on Windows 8, the free disk space gains are higher. Your Windows tablet is still going to take hours to install updates, and eventually use up a massive amount of space on your SSD. People complain about downtime and lost productivity but Microsoft does not value your time. The OS updating experience is slow, painful, disk bloating and the cleanup process remains equally slow and time wasting. It is far from the days of Windows XP, where your time wasn't wasted if you had downloaded all the patches and updates. They installed extremely quickly, and didn't waste your time at logoff, again at the next logon, and doing cleanup.

Basically the right design will be the one which minimizes I/O activity, not the one which increases it.


Anonymous said...

Finally, an article that truly describes the POS that is Windows Update.

I should add to that my experience of installing updates on a clean system. Besides having 2 GB of updates to install, Windows Update is leaking gigabytes of memory while installing them. I thought that four gigabytes of memory is enough and didn't have a swap file, and then tried to install 80 security updates and it went out of memory. And if the updates are successful, the memory is still not freed.

I must say that the solution is to not install clean Windows 7 but to install using the updated images kindly provided by software pirates :). That way I only had to quicky download 250 MB of updates without any problems.

Anonymous said...

Great article explaining things I never new, and in ways I could understand. Great job.

I completely agree about the time it takes updates to install in Vista+, but Vista and above are infinitely more secure than XP. I understand the article was written 5 years ago, as of this post, but, while XP was pretty snappy and updated faster, it was a swinging door for malware. Vista had a terrible launch, for many reasons, and I'm glad it's already gone, but it introduced security layers on par with linux and OSX that finally made it a modern OS with legitimate emphasis on data and system security. Anyone emphasizing to stick with XP, even 5 years ago, is irresponsible. Anyone on XP and using the Internet in any capacity, even in 2010 or 2011 before support was officially ended, was a ticking time bomb.

Gaurav Kale said...

Windows Embedded POSReady 2009 is just XPSP3 with a different branding and very minor changes. Its security updates can easily be installed on Windows XP Professional SP3. One should use whatever system he prefers. Microsoft should not get to force anything.