Why is separation between data and system files not a standard OS feature

I have a quick question, and maybe someone can enlighten me as to why things are the way they are. A good computing practice is to partition your drive into at least two parts. You put your data and settings on one partition, and your OS and software on the other one. The idea behind this is that you can re-install or upgrade the OS at any time without actually worrying about wiping out your data from the drive. This is nothing new, right?

In fact most software developers instinctively understand this problem. Most modern software has some limited support for the rare occasion when shit breaks and you need to remove it, and reinstall it from scratch. Its common for applications to store their cache, settings and saved states in some hidden directory in the users’ private profile. That directory does not get removed during routine un-installation unless you specifically request it, or go and delete it manually. This is commonplace.

Smart people go one step beyond this and put all that stuff on a separate partition. But it is tricky because the very process of partitioning drive is usually less than user friendly. Windows for example doesn’t even give you an option when you install it. Linux does, but it usually makes a big mess of the whole procedure. Most installers have two options:

  1. Dude, I don’t give a fuck – in which you let the installer do what it wants, which is usually create one big partition for / and one for swap
  2. Dude, I love partitioning drives – in which you get to do the tedious and boring busy work yourself

At least this was my experience while installing Ubuntu. And I’m using Ubuntu as an example, because this is as mainstream as you can possibly get with Linux. It’s the go-to distro starter for most people, largely due to the fact it is easy to install and configure. I haven’t really installed any Linux in the past year (I just upgrade existing boxes), so maybe things changed since then. The point is, that the “easy”, automatic installation option in just about every popular OS is almost always “put all my shit on a single partition”.

Why is that? Why can’t we build installers that partitions the drive automatically and separates data from system files by default? After all, most people are lazy and so they will instinctively gravitate toward the solution that requires least work. So why don’t we do some of the heavy lifting for them and set up their system the right way from the get-go?

I know, I know – there are some logistic problems here. For one, you probably don’t want to make decisions about your users’ hard drive. You don’t want them to wake up one day and go “I had a 300GB drive, but I can only see 200GB here – WTF?”. So maybe you could just present the user with a nice slider that let’s them pick between “I will be downloading a lot of pr0n and music” or “I install a lot of games and shit”. Moving the slider then adjust a nice pie chart that provides a much needed visual aid that explains the the ratio between “your shit” and “your computer’s shit” areas of the drive.

The second issue is detecting a data partition. If a user already has one in place, you want to leave it alone. This is a bit tricky to resolve, but I’m confident it’s basically a matter of standardization. Basically, you pick a partition and stick with it. So for your given OS, second (or third, or whatever) partition is always data. If the user originally installed the OS using the “I don’t give a fuck” preset that partition is guaranteed to be there. If they did it manually, then they can go fuck themselves and use the advanced installed features to figure that shit out. Once you standardize, it’s easy. Just assume data is there, and maybe ask user if he wants to wipe it or not.

My point is that this doable, but I don’t really see it being done by anyone anywhere. The only exception here is Chrome OS which takes this concept to the extreme, and moves your data off the hard drive and to the cloud. Chrome will be by far the easiest OS to maintain. If shit goes south, you can just re-image the whole drive on the spot. No backups, no troubleshooting, no bullshit.

We could easily get the traditional, non-cloud-based OS’s to a point where they can be restored just as easily. Press a button, wait 5 minutes, start with a fresh system and all your data intact. In fact a lot of hardware vendors already have this halfway implemented. Some Dell models for example have a hidden system recovery partition. You can boot into it, and it will run Norton Ghost and overwrite your drive with a factory default image. Of course this utility will wipe all your data in the process, so it is still considered a last-resort solution. But there is no reason why it needs to be like that. If implemented properly, this could evolve to be a non-destructive maintenance task that can be done few times a year to keep the machine in a good shape.

I’ve been wondering about this particular issue for a few years now. I’m secretly hoping that Chrome with it’s cloud based storage will finally push OS developers in the right direction and will make them start thinking about finally separating system and data files by default. Or maybe not.

This entry was posted in technology. Bookmark the permalink.



16 Responses to Why is separation between data and system files not a standard OS feature

  1. Mart SINGAPORE Mozilla Firefox Mac OS Terminalist says:

    But if you’re savvy enough to install your own OS, be it Windows or Linux, shouldn’t you also be knowledgeable enough to separate your system and data partitions? In effect, this would then be a moot point. It’s just PEBKAC.

    What I would like is for vendors to offer options on separating the Windows and data partition too. In one of your earlier posts, you wrote that Dell is offering buyers custom partitions of their harddrive. That’s a good step. This could extend to not only when buying online, but at brick-and-mortar stores as well.

    For OSX, well, I just set up Time Machine and then forget about all this. ;)

    Reply  |  Quote
  2. Randy UNITED STATES Mozilla Firefox Windows says:

    Well, if you’re using Ubuntu as the standard, then it’s important to realize that Ubuntu distributions, for some time, don’t remove /home directories on an upgrade installation: https://wiki.ubuntu.com/UbiquityPreserveHome

    I have to agree with the prior poster about installing and keeping partitions…

    Reply  |  Quote
  3. ink UNITED STATES Google Chrome Linux says:

    try pclinuxos ..the installer automatically creates two partitions, one for / and the other for /home

    this is not a function of linux as an operating system, it is a function of the installer and there are linux distro that have the behavior you want

    there are a lot of linux distro with a lot of different behaviors, if you use ubuntu and dont like the behavior it has, try another linux distro ..or ask for what distro has the behavior you want. Since not everybody can agree with everything, different distro allow different people to have different options

    Reply  |  Quote
  4. James Heaver UNITED STATES Google Chrome Mac OS says:

    This feels revolutionary but shortsighted at the same time. Do we still need partitions and hardrives in the same way at all?

    With the development of more types of storage, SSD, flash drives, cloud etc, we’re reaching a point where a computer should use multiple different storage mediums seamlessly.

    System files – on a super fast SSD
    MP3s and AVIs – on a super slow HDD, probably somewhere else on the network.
    Documents and emails – synced to the cloud (or edge) and available on every machine and logon you use, aswell as through a web interface.

    At this point the partitions become pointless. I want ‘My Pictures’ to be a folder view of Flickr. I want ‘My Documents’ to be a folder view of Google Docs, with everything available in ODT format so I can open in locally. I don’t want just my data partition mounted to /home, I want to pull bits of data from everywhere.

    Reply  |  Quote
  5. Zel FRANCE Mozilla Firefox Windows Terminalist says:

    I don’t know why it isn’t done yet, especially since both “My Documents” and “Program Files” folders can be set up to be on another partition. That would leave a partition for the system that can be restored without destroying user data (but then virus and/or malware may remain) and will always have enough free space for virtual memory (since you only have to account for Windows bloating, a couple dozen gigabytes is enough), and another partition for data that can be filled to the brim without the system becoming unstable.

    Anyway, it’s been a while since I installed Ubuntu or Debian, but it always defaulted to separate /, swap and /home.

    @ James Heaver:I like to know where my files are stored physically, at least for the sensible ones. I wouldn’t leave an important project on a USB disk or in the cloud, even if it’s only .doc or .odt files. I wouldn’t trust a seamless (read: hidden) storage solution at all.

    Reply  |  Quote
  6. SapientIdiot UNITED STATES Mozilla Linux Terminalist says:

    I think ubuntu is heading in the direction of seprating the home folder at least. With the most recent version your asked if you wish to encrypt home during installation, and when you choose this option it puts /home in a seperate partition… of course then the data is encrypted so i imagine it’d be lost on a reinstall anyway…. but its a start, it might actually use a seprate partion for home regardless of encryption, but i only have 1 ubuntu machine and its setup with the encyption option.

    ArchLinux also asks you to make a second partition for /home if you choose the IDGAF option, and if i recall correctly has done so for a while.

    I’m not sure about others, since almost all of my boxes run Arch. I would say Gentoo does too, but its pretty safe to assume anyone installing gentoo knows how to set things up the way they like. Fedora uses an LVM by default, and i’m not quite sure if they seperate home or not.

    Reply  |  Quote
  7. I think Ubuntu has just dumbed down everything too much – Mandriva and Sidux for example create separate root and home partitions by default (the way it should be), and also have very a user friendly custom partitioning option in the installer. You can simply select the data partition and specify /home as the mont point.

    Reply  |  Quote
  8. Morty NORWAY Konqueror Linux says:

    At least this was my experience while installing Ubuntu.

    Yes, and as far as i can tell it’s about the only distribution doing so. So your line of arguing kind of falls trough. If you look at desktop Linux installation base world wide, you see the majority do not do this. If you actually look at the big installation and tone down the Ubuntu hype to realistic proportions, you will see sane behavior from the other major distributions(Wich together have a far greater install base than Ubuntu). Like (Open)Suse, Mandriva not to forget Red-Flag.

    Reply  |  Quote
  9. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    @ Mart:

    Yes, that’s true. The problem is, hardly anyone is doing it. When you set up machine for a friend or relative do you go through all this trouble or do you just slap on the OS with all the default settings, install antivirus and call it a day? Most people do the latter rather than the former.

    Defaults are a powerful thing – if this was a default feature, everyone would be using it without thinking about it to much.

    @ Randy:

    Nice, I didn’t realize that. I usually format drive between test installs, and merely upgrade my main ‘buntu box without reinstalling.

    @ James Heaver:

    Wow, great idea. I’d love to see more machines being built with several dedicated drives. Super fast SD drive for system files, another one for programs, large capacity traditional HD for data, etc..

    One thing about cloud content – this does pose some security concerns. For example, where do I put my pr0n collection? Or for example “private” pictures of significant other – ones that you wouldn’t want floating online, even if they are supposedly secure behind login screen and encrypted. Or your pirated music and movies?

    Also, having your data live in the cloud would introduce a lot of latency. I’d ideally want the data to be mirrored locally, and synced with the cloud the way getdropbox.com does it. Otherwise, having your system files on super-fast SD drive won’t mean anything if your data is on a super-slow network drive. Working with your files will feel sluggish.

    @ Morty:

    I used Ubuntu as an example because it is one of the few linux distros available as OEM. You can for example buy a Dell machine with Ubuntu on it, and sure as hell it will have two partitions: / and swap, because that’s the default.

    This basically makes Ubuntu the most common “beginner” distro. Seasoned Linux users can and will reach for different Linux brands, but they usually have enough know-how to set up their partitions correctly to begin with. An average computer user (ie. the silent majority) on the other hand will buy a machine with Windows, OSX or Ubuntu and all of them will have a single partition setup.

    I’m basically saying that these particular systems could really use a multi-partition default, because it would make it the de-facto standard for MOST computer users.

    Reply  |  Quote
  10. James Heaver UNITED STATES Google Chrome Mac OS says:

    @ Luke Maciak:

    Thats why I generally prefer an edge solution to a cloud solution.

    I don’t trust data in the cloud. From examples such as the sidekick dataloss to a uk example where the voicemails of several royals and celebrities were hacked into by a newspaper.

    I would store my data my local network and have it accesible over the internet.

    You would need some way of tracking how secure a document was. If you’re integrating with flickr etc then you would already need to highlight which are public and which aren’t

    @Zel
    I can see your point about knowing where things are, but part of this would involve duplicated storage and backup. Of course everything would be stored on your slow HD along with wherever else it’s stored.

    There are also cloud based solutions that are secure. These distributed, encrypted schemes. Again, this feeds into the edge based ideal of things.

    Reply  |  Quote
  11. copperfish Mozilla Firefox Ubuntu Linux Terminalist says:

    If you can clearly partition off only data, then it makes sense.

    The problem with current versions of Windows, Linux and (I assume) OSX et al. is that /home or “My Documents” is a dumping ground for anything but data. Application preferences and data sit everywhere.

    So you keep your home partition – great. You keep your data and documents – great. But stuff like saved games and application preferences get overwritten when you install an application again – not so great. Worse, the average user has no idea those hidden configuration files are there.

    This will remain a painful manual process into the future until we get some kind of portable application preferences XML standard or something similar. Install Shield and DEB/RPM packagers are you listening?

    Reply  |  Quote
  12. Matt` UNITED KINGDOM Mozilla Firefox Windows Terminalist says:

    I have the whole “Documents and Settings” folder on a separate partition, which mostly works well, but occasionally doesn’t when I forget that something’s being stored on the C: drive with the OS and programs.

    For me the ideal system would let me nuke the OS whenever I want with nothing else in the firing line, so have 3 partitions – OS, Programs and Data. But there’s always issues with the fact that “Program Files” defaults to C: (I’ve yet to find an easy way of changing that, in the same vein as how you can fiddle with the registry to move your personal folders)

    Then programs continue to have “C:/Program Files” hardwired as their install location, reinstalling Windows blanks out the whole “Add/Remove Programs” list because the programs haven’t been installed to that install of Windows – they still work, but you lose the utility of a collected list of what you have installed. The whole thing just gets annoying so I stick with the OS and programs being together. With the unfortunate side effect that any settings or data that get stored under the program folder will get wiped if I reinstall the OS.

    Reply  |  Quote
  13. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    @ copperfish:

    Actually, most linux distros will keep most of “application settings” in your home directory. For example your Firefox profile is in ~/.mozilla folder. Most of your personalized KDE settings sit in ~/.kde and etc…

    System wide settings are usually somewhere in /etc but anything pertaining to userland apps or your particular user account sit in ~.

    This is a problem with Windows though seeing how most XP era games loved to keep saves in the game folder inside C:\Program Files. Vista curbed this a bit by requiring privilege escalation to write to Program Files. Newer games tend to put their saves somewhere in C:\Users.

    Also Vista sort of changed the focus from My Documents to the user home directory. Standard folders like Pictures, Music and Video are now outside Documents folder. Therefore the implication is that if you want to back up your documents and settings you should back up the whole C:\Users\username folder rather than Documents alone.

    This is much better than the XP way in which My Documents is a special shortcut on desktop and in the Start Menu and the purpose of the Documents and Settings is something most users remain blisfully ignorant about.

    @ Matt`:

    Yep. But as I said above, new applications written in the age of Vista and Windows 7 are much more aware of this issue and tend to store their data where they should – in the home directory. It’s either that, or require users to run them as Admin which is a great way to annoy the hell out of your user base.

    Here are few examples – on Vista Bioshock saves games in:

    C:\Users\username\AppData\Roaming\Bioshock

    Assasin’s Creed saves in:

    C:\Users\username\AppData\Roaming\Ubisoft\AssasinsCreed

    Bioware games now save in:

    C:\Users\username\Documents\BioWare

    So this trend is changing.

    Reply  |  Quote
  14. Luke Maciak UNITED STATES Mozilla Firefox Windows Terminalist says:

    @ James Heaver:

    Thanks for the link to the edge talk – too bad it cuts off halfway through. I read O’Brien’s whole rant, and I really enjoyed it.

    To me the main issue with the edge is reliability. It’s the vacation syndrome – ie. your home server dies 3 days after you leave for a 3 week vacation.

    Actually, I’d fucking love to take a 3 week vacation, but unfortunately hardly anyone on this side of the big pond has that many vacation days in a year to begin with. That’s the thing I really miss about living in Europe – vacations were proper length there. When I was a kid, my family would spend at minimum a month out of the year vacationing at the Baltic sea. But I digress…

    What I personally like is the combination of Edge and Cloud technology. Stuff like getdropbox.com actually get this stuff right. With Dropbox I have a local cache of my files on every computer I bind to my account. If I lose one of the nodes, I can restore it from another one. And if I don’t have connection to the cloud, I still have access to all my data everywhere that I used it.

    Reply  |  Quote
  15. IceBrain PORTUGAL Mozilla Firefox Windows Terminalist says:

    James Heaver wrote:

    This feels revolutionary but shortsighted at the same time. Do we still need partitions and hardrives in the same way at all?
    With the development of more types of storage, SSD, flash drives, cloud etc, we’re reaching a point where a computer should use multiple different storage mediums seamlessly.
    System files – on a super fast SSD
    MP3s and AVIs – on a super slow HDD, probably somewhere else on the network.
    Documents and emails – synced to the cloud (or edge) and available on every machine and logon you use, aswell as through a web interface.
    At this point the partitions become pointless. I want ‘My Pictures’ to be a folder view of Flickr. I want ‘My Documents’ to be a folder view of Google Docs, with everything available in ODT format so I can open in locally. I don’t want just my data partition mounted to /home, I want to pull bits of data from everywhere.

    You can have mounted dirs inside other monted dirs – So you can have /home in one partition, and you can mount FlickrFS in /home/username/photos, and GooFS for your /home/username/documents.

    Have I told you I love Fuse, and it’s filesystems? :)
    Curlftpfs is especially useful.

    Reply  |  Quote
  16. Andrew Zimmerman UNITED STATES Google Chrome Windows Terminalist says:

    I really liked the efficiency of the ZFS and we’ll see where these times lead us to filesystem management. Your idea about eliminating the Hierarchy standard file management sounds appealing. I think our future lies in the metadata, and if we manage that well along with the underlying FS then we’re fine, or in the clear. Hopefully now.

    Reply  |  Quote

Leave a Reply

Your email address will not be published. Required fields are marked *