A lesson learned I believe worth passing on

While looking for backup software last year I was pointed towards Genie Timeline (GTL). Tried the free version then bought it for both my PCs. It was just what I wanted and have suggested it to others. One friend tried it, liked it, and bought two copies for her home XP PC and her work Windows 7 laptop. She also bought two portable external drives.

In Disaster Recovery mode GTL worked fine on the XP machine but on the Windows 7 it never seemed to complete. It would seemingly archive everything, then find another 3,000 items. It processed these though not apparently archiving them, then find another 3,000, and so one. And so it never completed.

When I had a look I tried various things but couldn't see what was failing. It did look as if the 3,000 chunks of files were mostly, if not all, in the C:\Windows\Winsxs folder. I changed the backup type to non-DR and that worked fine.

I logged a ticket and went through various diagnostic sessions - difficult as I could only get the laptop occasionally. Eventually a remote access session was set up for Wednesday this week for Genie to investigate. After about 90 minutes the tech.support person was able to find out that GTL was getting an error when it tried to create a folder on the external drive and asked me to CHKDSK it and possibly to reformat it.

I then tried to manually create a folder on the drive and I too got an error. Googling this error I found this link http://help.lockergnome.com/windows2/file-folder-lim... which includes

In reference to help.lockergnome.com/windows2/file-folder-limits--ftopict450749.html:

A FAT32 directory can have 65,536 directory entries. Each file and subdirectory takes from two to thirteen entries, depending on the length of its name, so those entries can disappear long before you think you've used them all up. Your total of 22,657 files could very easily use 65,000 entries.

So, given the number of files and folders in C:\Windows\Winsxs and the length of the folder and file names, FAT-32 would almost certainly have a problem holding this folder which, on the Win 7 PC, has in excess of 15,000 folders and a total of over 60,000 files. This I suggest was probably the root cause of the problem. Winsxs seems to be new in Vista and Windows 7, XP doesn�t have the folder so doesn't get the problem.

I tried to Convert the drive to NTFS, it failed after 15 minutes or so saying it was dirty. I tried CHKDSK /F which reported no errors. I copied the data from it (3 hours or more) and then tried reformatting it, it failed after several hours with no explanation. After trying a few other things I went into Disk Management from My Computer / Manage. Did Delete Volume, then New Volume (or similar), then Format NTFS. That worked!

Copied the data back on - another 3+ hours. Run up Timeline and this time it successfully completely ran a Disaster Recovery Timeline session. Hooray!

So the original problem was caused by the drive being formated in FAT-32 format, as many portable drives still seem to be, and GTL not fully recognising the situation and aborting or blacklisting or whatever the Winsxs folder.

Thinking further about the inability to NTFS convert or format I suspect the drive had a hidden partition which I understand can cause that sort of problem.

As often the case in hindsight it's obvious what the problem was. Nevertheless I feel it worth passing this on as others may have an analogous problem with very large folders, especially Vista or Windows 7, and FAT-32 drives.

First thing I did was format my external drive to NTFS because of the file size limit in FAT32 of 2gb. My current backup solution Paragon Backup and Restore uses 4gb files to store images which can be changed but the recommendation was to use NTFS.

As did I, however the average person with a home computer would not know what a file system let along the pros and cons of the different ones. Why are manufacturers still formatting their new drives in FAT-32? Is it to do with MAC compatibility?

Paucity of choice I think. FAT32 may well be patent free with no fee required for its use.

Ah, I didn't know that. It still begs the question of how the average user is supposed to cope with such things.

I don't believe that you have to pay a patent fee to format a drive. I suspect that it is just down to using the most widely applicable format.

Just one small correction. The maximum file size on a FAT32 volume is 4GB - 1 byte.

I stand corrected, well sit.

The average user isn't. Writers of software for payment should be aware of the limitations though.

In reply to a post by cheshire_man:

So, given the number of files and folders in C:\Windows\Winsxs

Eventually WinSxS is going to eat up the whole hard drive. It grows with every Win Update. Yet there is no way of thinning it down on Vista or 7, unlike XP where you can generally safely get rid of the equivalent $UnInstall folders after a month or two.

My winsxs directory got up to 13GB before I reinstalled (Win7) and it's already at 6GB. Maybe they'll fix it for Win8...

or 9...

or 10

Fix what?

In reference to blogs.technet.com/b/askcore/archive/2008/09/17/what-is-the-winsxs-d...:

All of the components in the operating system are found in the WinSxS folder � in fact we call this location the component store. Each component has a unique name that includes the version, language, and processor architecture that it was built for. The WinSxS folder is the only location that the component is found on the system, all other instances of the files that you see on the system are �projected� by hard linking from the component store. Let me repeat that last point � there is only one instance (or full data copy) of each version of each file in the OS, and that instance is located in the WinSxS folder.

Fix the fact that you cannot get rid of the crud that is stored there. You don't really think that Windows needs everything stored in that directory, do you?

There's a lot of misunderstanding about winsxs, as BatBoy points out. I guess that Windows users aren't really up to speed on the concept of hard links or symbolic links.

I think you should buy a bigger HD tbh.

In reply to a post by cheshire_man:

As did I, however the average person with a home computer would not know what a file system let along the pros and cons of the different ones. Why are manufacturers still formatting their new drives in FAT-32? Is it to do with MAC compatibility?

One of the reasons. I don't use any removable media without formatting it as NTFS first.

FAT32 is not worth the risk, or trouble!

MS have stated that they believe the disk consumption used by winsxs is worth what benefits it gives. I'm inclined to agree with them tbh.

It's adaptive afterall, if you don't install loads of [censored] it doesn't grow as much. I'm not singling you out, my winsxs folder is fairly large, but it's there to help me go about my business on the PC.

When you see computers where it has grown very large there are always underlying problems with the computer, malware, junk installations etc.

In reply to a post by BatBoy:

I think you should buy a bigger HD tbh.

I've just got myself a 128GB SSD for my C: drive. It's already half-full and I haven't installed that much software. So much for Win 7 only needing a 16GB disk drive.

Edited by micksharpe (Sat 17-Mar-12 00:39:55)

In reply to a post by Pipexer:

MS have stated that they believe the disk consumption used by winsxs is worth what benefits it gives.

Well, they would say that, wouldn't they?

In reply to a post by Pipexer:

It's adaptive afterall, if you don't install loads of [censored] it doesn't grow as much.

As I understand, it will grow with every Windows update that is installed.

I know a kludge when I see one and Windows SxS is a kludge imo.

In reply to a post by BatBoy:

The WinSxS folder is the only location that the component is found on the system, all other instances of the files that you see on the system are �projected� by hard linking from the component store.

I'm not convinced by that, for instance:

Directory of C:\WINDOWS\System32

12/04/2011 16:07 892,416 kernel32.dll

vs.

Directory of C:\WINDOWS\winsxs\x86_microsoft-windows-kernel32_31bf3856ad364e35_6.0.6002.18449_none_9582275d538a1db6

12/04/2011 16:07 892,416 kernel32.dll

If they were linked one of them would show as a <JUNCTION>.

If they were (are) hard links? http://msdn.microsoft.com/en-us/library/windows/desk...

Hard links don't show up as anything special in a directory listing. They certainly wouldn't pretend to be Junctions.

Thanks! Is there any way of confirming they are hard links?

I don't know of any easy way to display hard links. Perhaps someone clever than me knows the answer to that one.

You mean apart from the text I quoted stating they are hard links?

fsutil hardlink list filename

That's right! Confirmation not hearsay!

'fsutil hardlink' does not support 'list', only 'create' on my Vista.

Hearsay from the dev team?

http://technet.microsoft.com/en-us/query/hh290814

C:\>findlinks C:\WINDOWS\System32\kernel32.dll

FindLinks v1.0 - Locate file hard links
Copyright (C) 2011 Mark Russinovich
Sysinternals - www.sysinternals.com

c:\windows\system32\kernel32.dll
Index: 0x0000BC13
Links: 1

Linking file:
c:\Windows\winsxs\x86_microsoft-windows-kernel32_31bf3856ad364e35_6.1.7601.17651_none_95971084b4b0c29f\kernel32.dll

C:\>

Nice! Thanks.

I guess the fsutil hardlink list command would work on Windows 7. It did work on the Windows 8 consumer preview I had.

C:\>fsutil hardlink list C:\WINDOWS\System32\kernel32.dll
\Windows\winsxs\x86_microsoft-windows-kernel32_31bf3856ad364e35_6.1.7601.17651_none_95971084b4b0c29f\kernel32.dll
\Windows\System32\kernel32.dll

C:\>

Windows 7?

Yes

OK.

Not understanding the significance of this thread I asked my XP tower to look in my NTFS files & folders for WinSxS --- it produced same spread across 78 folders holding 341 files totalling 66MB. Should I feel deprived?

WinSxS is more of a "problem" with Windows Vista and above. I say "problem" in quotation because usually it goes astronomically large when the user has done something foolish like got a virus or installed some software which has broken things.

Brand new machine out the box and it's 7.5gb. Interesting. Doesn't really bother me though as long as it's not 50gb +

Post deleted by billford

interesting on the fat32 limits, its good you mentioned this.

This is where the HP USB Disk Storage Format Tool is invaluable.

Post deleted by billford