FAT32 Limitations
I bought a 320 GB external Lacie drive specifically for the purpose of backing up my Windows box. Yesterday I tried to push a 32GB Windows backup archive onto it, and I was told that I don’t have space on the drive for that file. This was of course not true, because the drive was mostly empty with only few small files I copied over the previous day and a 50 GB ghost image split into multiple 3GB files. There was plenty of space for my file on the drive.
Then it dawned on me - I quickly popped over to the Management Console and confirmed it. The disk was formated as FAT32. This is not a bad idea - FAT32 is a viable choice, and most of your USB sticks and memory cards these days still use that file system. Unfortunately it has one important drawback: a maximum file size limit of 4 GB. This essentially makes it useless as a backup drive.
So I converted it to NTFS using the built in WinXP tool:
convert J: /FS:NTFS
NTFS, lime most of the modern file systems do not have any file size limitations. Unfortunately it is a proprietary system and writing to it is still shaky under Linux.
Which brings me to this point: if I wanted to format my drive using a modern file system with no file size limits and journaling which one would I choose? The answer is really simple if you limit yourself to a single OS:
- If you use Windows, choose NTFS
- If you use Linux choose ext3, raiser or etc..
- If you use OSX choose HFS+
- etc..
But what do you do, if you own all 3 platforms and you want seamlessly exchange data between them using an external drive of some sort? Most of the smaller USB drives are routinely formated in FAT32 because it is the most universal system out there. But it is not the most optimal solution - and the file size limit will render it downright useless in a few years.
I know that historically file systems were closely tied to the OS. I’m not saying we should keep the two separate. I just think that we need a modern OS agnostic, journaling file system for data exchange devices such as flash drives, pocket hard drives and etc.
And yes, I know that there are always workarounds. For example, I could use captive NTFS in Linux. Or I could simply connect the drive to a windows box, and share it via network allowing Linux and Apple clients connect via samba protocol.
But wouldn’t it be nice to have a single standardized data exchange file system? We could build it with journaling, and robust redundancy and error checking and make sure it is platform independent and easily portable. Is anyone working on something like that? Would it even be feasible to propose such a project in the current software climate?
Related Posts:
March 12th, 2007 at 2:51 pm (3311) [Quote]
Yes, that would be awesome - then the main obstacle between me and an easy linux/windows dual boot would go away
I want to keep the files for both systems on the same partition, but trying to share an ntfs volume has the issue of linux support being flaky, and trying to share an ext3 volume (using an ext2 driver for Windows) won’t work either, basically because of differences in how the 2 OSs work with the FS, FAT32 has exactly the problem you described - limits on sizes
If there was one, fully up to speed file system that was as widely supported as FAT, then the whole issue could have been avoided
Somehow I can’t see it happening any time soon though - Microsoft seems to always want to supersede the existing standards with its own weird take-offs
Posted usingMarch 12th, 2007 at 3:07 pm (3312) [Quote]
Yeah, MS is working on WinFS which will be a “relational” file system widely different from the ones used at the moment. Chances are we will not see a solid Linux and Apple support for that format for years to come, unless MS decides to open it up a little bit.
So an universal file system definitely won’t come from MS.
Posted usingMarch 12th, 2007 at 5:15 pm (3313) [Quote]
I ran into the same issue, only I have even more complications because I run Solaris! (ufs or zfs).
My solution is one people don’t think much about . . . a file server!
I found this cool disk enclosure at CompUSA. It has both SMB and NFS, making it perfect for all systems. I configured it with DHCP. This works perfectly when connected direct to a Windows box with a crossover cable. NetBIOS will always find the box via broadcast. I dropped a 300GB disk in the chassis and it all works like a dream. It is also just as portable as any external USB drive.
Now, if you have one of those off-brand NICs, you might have issues with drivers for things like Ghost, but this is a good alternative.
Posted usingMarch 12th, 2007 at 7:01 pm (3317) [Quote]
Good idea. I was thinking about something like that too. How does the transfer speed using a crossover cable compare to a USB 2.0?
Posted usingMarch 12th, 2007 at 7:19 pm (3318) [Quote]
Not sure actually. Network seems faster than USB. I will try to do some benchmarking for you.
Posted usingApril 7th, 2007 at 9:01 am (3934) [Quote]
Well. OS X, Linux are now working on to bring in ZFS to their native supported file systems. Solaris have has ZFS for a while now. So we see some steps toward a universal file system. We should wait to see what MS is going to do about it. MS’s new relational WinFS shall be really good and beat ZFS to have any chance of staying. I don’t know much about it, but can’t think of any FS feature not already in ZFS.
For now, I use FAT32 for external drives to be used on all my systems (MAC, SUN, PC) and created my own custom backup/restore scripts, which breaks down large files ( > 4GB) into peaces during backup (adding an special prefix), and re-assemble them during restore.
Posted usingApril 7th, 2007 at 1:28 pm (3940) [Quote]
It would be great if ZFS could be the next big thing. But is Microsoft going to implement it? Of course not. They actually benefit from not having a uniform exchange format like that.
Posted usingApril 8th, 2007 at 7:57 am (3960) [Quote]
Unless they try and pull an “embrace, extend, exterminate” on it
Posted usingMay 29th, 2007 at 1:02 am (4612) [Quote]
Does anyone have the patience to expand on the two sentences below from Craig Betts?
Please provide a more detailed explanation for the benefit of an old-timer who has apparently been left somewhat behind. The help is for me to duplicate the approach.
“It has both SMB and NFS, making it perfect for all systems.
I configured it with DHCP.”
Setting up a RAID server with four 250 GB drives with a mobile rack to switch OS without resorting to dual boot. Search for universal format among LINUX, Windows, Etc. led to this discussion.
Posted usingMay 29th, 2007 at 1:13 am (4613) [Quote]
I think he simply means that the enclosure has some sort of on-board network card which lets you share an NTFS volume using NetBios/Samba which makes it easy to work with for both windows and linux.
I wrote about mounting windows shares in linux before.
Posted using